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ABSTRACT 

This guide is designed to increase a school's 
productivity by presenting some simple yet powerful techniques for 
looking at the school and making decisions to address the most 
immediate problems. Tracking success is tracking progress and using 
evaluation as a tool for implementing changes from school-based 
management to complete restructuring. Chapters, which review the 
basic steps of evaluation, are: (1) "Introduction. Evaluation--A 
Management Tool"; (2) "Step One: Focus the Evaluation"; (3) Step 
Two: Identify Tracking Strategies"; (4) "Step Three: Manage 
instrument Development and Data Collection"; (5) "Step Four: Scoring 
and Summarizing Data"; (6) "Step Five: Analyze Information and 
Interpret Findings"; (7) "Step Six: Use Findings and Contin Program 
Moiitoring"; and (8) "Reprise and Revelation." Five resources contain 
sample forms for evaluation data collection, including a sample 
school report card. Seventeen tables and one figure summarize 
evaluation techniques, and there are six sample worksheets. Suggested 
readings (35 in all) follow each chapter. (SLD) 
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Introduction 

Evaluation— A Management Tool 



New Demands 
on the Schools 

The demand for 
increased 
productivity. 



We educators are living in a time of decreasing resources and increas- 
ing demands both from the communities we serve and from the wider 
body politic. Our morning newspaper regularly reproaches us about 
the declining quality of American education, the unfavorable stand- 
ing of American students on international assessments in mathemat- 
ics and science, and the poor productivity of our schools. Each baleful 
article includes test scores and statistics to boki-C? its incontrovert- 
ible assertions. State legislatures, in an attempt to finance increasing 
public needs with shrinking revenues, e-'Vrt us to become more 
productive and accountable and legislate ^nutions such as "choice," 
"restructuring," or "school-based manag. n^nt" as ways to leverage 
the educational dollar. It is no wonder that those of us still optimistic 
enough to seek out the challenges of educational leadership ask 
orrselves with each new school improvement initiative, "Don't I 
already have enough to do?" 

The answer: Yes! And the solution for those of us implementing 
new accountability, restructuring, and improvement mandates who 
wish to remain sane is to do with our time what legislators would have 
us do with their monies— get more "bang for the buck." In industry, 
increasing productivity generally means producing a larger quantity 
of goods and services using the same amount of resources. In educa- 
tion, productivity is not so easily defined. Increasing our productivity 
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implies improving the quality of our processes and outcomes while 
tailoring our services to an increasing number of specialized clients 
(including limited English speaking, physically or educationally hand- 
icapped, developmentally delayed, and children of working parents). 
Unlike industry, our business is people ii tensive; changes are medi- 
ated by the attitudes and skills of our staffs. Educational productivity 
cannot simply be mandated; nor can we be mechanized. Teaching 
machines, film projectors, and computers (for the most part) have not 
changed the way our classrooms look and operate. On the other hand, 
powerful ideas and research-based instructional strategies such as 
cooperative/collaborative learning, process-based writing instruc- 
tion, and meaning-centered mathematics instruction have. For those 
of us who already have more than enough to do, doing more with even 
less is only possible if we acquire new techniques that allow us to 
leverage our already limited time into strategies for meeting new 
mandates and challenges. 

Help for the Tracking Your School's Success: A Guide to Sensible Evaluation 
beleaguered, is designed to increase your productivity by presenting some simple, 
yet powerful techniques for looking at your school and making deci- 
sions that will address its most immediate problems. Tracking your 
success is simply tracking your progress. This book provides tech- 
niques for tracking where you are going, monitoring whore you are, 
and identifying strategies for getting you where you want to be. 
Tracking your success means using evaluation as a tool for imple- 
menting changes from school-based management, to a better account- 
ability system, to complete restructuring. Tracking your success also 
means that, by using evaluation techniques to monitor your school, 
each year's agenda will be more focused and the basis for your decision 
making clearer (and fairer) to all in your school community. You will 
know how your efforts relate to school outcomes and will be able to 
use this information to create a sensible plan for school improvement. 



Why Evaluate? 

A switch from School-based management and restructuring, by placing more 
informal to formal authority and responsibility at the school level, shift accountability 
decision making, from the district to the site. Suddenly, constituencies who used to be 

referred "downtown* for information must be addressed by site man- 
agers and staff. This shift in accountability forces school staff to 
formalize and make public both their decision-making processes and 
the information upon which their decisions are based. Local evalua- 
tion is a powerful t^ol that supports this process. 

The challenge is for educators to configure evaluation strategies 
sensitive to the unique needs and priorities of individual schools — 
strategies that will inform the goals educators are trying to achieve 
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and the processes they are endeavoring to implement that, at the 
same time, will be credible to ever more skeptical audiences. The 
opportunity lies in the fact that school-based management and school 
restructuring place authority in the hands of those who have always 
been held accountable for educational outcomes: teachers and princi- 
pals. When both power and responsibility reside in the same institu- 
tion, a unique climate is created, one in which evaluation and assess- 
ment efforts can actually stimulate school improvement. 

Small change, The new demand for schools to adopt an explicit and documented 
large payoff, decision-making process can yield unexpected returns. First, a formal, 
public system of tracking your progress can assure parents that their 
chosen school is dedica* r.*I to meeting the needs of their children. The 
school community widens each year and the public's appetite for 
information seems to Le growing. School- Wei progress tracking has 
even been mandated in some states. In California, for example, 
legislation guaranteeing minimal levels of school funding requires all 
schools to issue an annual accountability "report card" to any and all 
interested people at least once a year. Among the areas covered by 
such report cards are the following: 

• curriculum 

• instruction 

• parent-community relations 

• school climate 

• stalling 

• student outcomes 

For schools operating under similar accountability mandates, the re- 
port card can provide a starting point for school-level evaluation and 
program improvement. Customized to your school's unique goals, such 
report cards can be a powerful strategy for heralding your successes. 
Table LI provides an example of a report card for a high school. 
Resource E has examples of report cards for an elementary school and 
a middle school. 

Relevance. Second, evaluation conducted at the school level can improve the 
quality of information used to make management, program, and 
instructional decisions. Who among us has not wondered what effect 
the implementation of whole language, literature-based language 
arts instruction or the elimination of "selective" criteria for entrance 
into advanced placement classes would have on politically important 
indicators such as state test scores or college admission? A formal, or 
more explicit, evaluation process provides information about new pro- 
grams and practices that are credible to both advocates and skeptics. 
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TABLE 1.1 Excerpt From Coastal Village High School Report Card, 
May 1991 



School Profile 

Founded in 1955, Coastal Vil- 
lage serves 3,150 students in 
grades 9-12. Our cosmopolitan 
student body represents over 
40 languages and cultures. Cur- 
rently, 17% of or*" students are 
classified as limited English 
proficient. Our ethnic mix is 
25% Hispanic, 18% Asian, 7% 
African American, 10% Pacific 
Islande/, and 40% white. We 
have 20% of our student body 
enrolled in gifted education pro- 
grams and 16% served by spe- 
cial education. 

Staff 

Of our 75 teachers, over 50% 
have master's degrees and two 
hold doctorates. Our specialized 
staff includes a librarian, nurse, 
activities dire* ! >r, athletic direc- 
tor, four counselors, and a com- 
puter specialist. Administration 
includes a principal and three as- 
sistants, one each in charge of 
counseling, curriculum, and dis- 
cipline/athletics. 

Professional development activ- 
ities included six teachers at- 
tending the national mathemat- 
ics teachers' conference; two, 
the social studies; and eight, 
the English. The county pro- 
vided training in collaborative 
learning, integrating social 
studies ar<d the arts, and 
peformance assessment, We 
also had required training in 
CPR, child abuse reporting, 
and earthquake preparedness. 

Curriculum and Instruction 

Students are required to take 
six classes and need 250 cred- 
its to graduate. Social studies 
and English a^e required for 
four years, mathematics and 



science for three. The elective 
program is extensive and fea- 
tures a robotics program and 
an arts "school within a 
school.* This year, we are offer 
ing Japanese and Russian, sta- 
tistics, and sculpture for the 
first time as a result of student 
interest. A second innovation is 
the substitution of an inte- 
grated English-social studies- 
art history humanities core in 
place of the traditional two pe- 
riods of English and history. 
Other curriculum changes in- 
clude the elimination of 
"tracked" classes and screening 
tests for advanced placement 
classes. 

Student Outcomes 

About 40% of our seniors take 
the SAT. Scores for the past 
year were Verbal, 472, and 
Math, 501. There were no large 
differences in performance be- 
tween boys and girls. About 70% 
of our students enroll in higher 
education: 15% at the state uni- 
versity, 25% f t state colleges, 
and 30% at two-year institu- 
tions. About 7% join the mili- 
tary. The average freshman 
GPA for our students attending 
universities is 2.7; it is 2.9 for 
those in community colleges. 

Attendance averages 85% with 
February being the month with 
poorest attendance. Our dropout 
rate is 4%, evenly distributed 
among Hispanics, whites, and 
Pacific Islanders. 

Last year five students won Na- 
tional Merit Scholarships, four 
won art or music contests, two 
were finalists in the National 
Geography Bee, and our Math 
Team came in third in the state. 
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TABLE 1.1 Continued 



School Climate 

Students are expected to main- 
tain appropriate behavior and 
sign the honor cods each Sep- 
tember. Violations result in de- 
tention. We provide many op- 
portunities for student 
recognition, which include the 
Trip to Washington for Good 
Citizenship, Student of the 
Month, Principal's Honor Roll, 
student government, perform- 
ing arts ensembles, and intra- 
mural athletics. 

Parent Involvement 

Senior citizens donate over 100 
hours a week as classroom 
aides, and parents have raised 
nearly $2,000 through Booster 



Clubs and magazine drives to 
fund our annual court house 
visit and Washington trips. 
Parent volunteers serve as 
translators during LEP screen- 
ing and in sheltered classrooms 
on an as-needed basis. 

Budget 

We receive $3,231 per student 
in state funding, which is sup- 
plemented by student fees for 
parking, athletics, and materi- 
als as well as parent gifts (to- 
taling over $7,500 last year). 
The average teacher salary is 
$34,000 (about 83% of the bud- 
get) and the principal earns 
$60,000. Administrative costs 
are 8.3% of the budget. 



When we have limited resources, decisions about which programs are 
worthwhile require more stringent criteria than "the kids like it" or 
"teachers say it's a great program." We are also operating in a climate 
of public dissatisfaction with the schools in general (though polls 
show parents feel their own schools are exemplary). Thus new pro- 
grams are nearly always instituted with less than full support. Par- 
ents welcome change but not change that makes school different than 
those they remember: Where are those worksheets? Why is my child 
playing with beans, blocks, and an abacus instead of learning the 
times tables? How can my child possibly write without being able to 
identify gerunds? Information about program outcomes can calm 
these concerns or help you improve newly adopted curriculum to 
incorporate the benefits of former approaches into current ones. 

Third, school-level evaluation activities provide parents, teach- 
ers, principals, and even district administrators with concrete evi- 
dence of school improvement or quality in a timely manner. Data 
gathered at the site, unlike data gathered during district or state 
assessment and evaluation programs, are manageable in quantity, 
generally easily understood (they are not subject to the esoteric trans- 
formations characteristic of large-scale testing programs), related to 
local questions and concerns and summarizable in a reasonable 
period of time. Timeliness and relevance are perhaps two of the best 
reasons for engaging in local evaluation. 
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Broadens the 
school's perspective. 



Finally, school-based evaluation reinforces a school-wide vision. 
A parent's primary concern is with what happens to the child. A 
teacher's foremost concern often is with the group of children being 
taught. Evaluation provides a framework for engaging teachers, par- 
ents, and the wider school community in the improvement process as 
well as a mechanism for informing all involved of school successes and 
strengths. This larger perspective helps build support for the local 
school as well as increased mutual understanding among different 
members of the school community. 



What Is 
School-Based 
Evaluation? 

Tracking your 
success. 



School people— principals, counselors, program coordinators, 
teachers, and aides — are constant, often unconscious, evaluators. In 
the course of their daily work, school staff make decisions based on 
informal data-gathering strategies and then assess the adequacy of 
those decisions with further data, both formal and informal. 

• Principals use district-level enrollment projections, informal 
reports from teachers, and inquiries from parents to determine 
staffing and classroom assignments for the next school year. 

• School and district administrators plan staff development pro- 
grams based on needs ascertained through observing informal 
classroom activities, monitoring new educational trends, and 
determining whether local school conditions are appropriate 
for trying innovations as well as listening to teacher concerns 
in the faculty room. 

• Administrators and teachers carefully monitor standardized 
test scores to see whether schools are "measuring up" and to 
determine whether any problems are surfacing in the instruc- 
tional program. 

• Teachers and principals use Back-to-School and parent nights 
for feedback about parent concerns. 

• Many high schools track their success by monitoring what 
happens to graduates (and those who don't graduate) and how 
students stack up on such indicators as number of academic 
courses taken, college freshman GPA, and number of students 
with chronic absence problems . 

• Administrators often compare their schools with others they 
deem similar by using such common measures as number of 
awards received, years of accreditation granted, special grants 
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received, Academic Decathlon performance, test scores, and, of 
course, athletic record. 

Those of us who are teaching or who have taught are adept 
"progress trackers. " We become "kid watchers," gathering data about 
student attitudes and preferences, classroom performance, written 
and oral performance, and formal grades on quizzes, tests, and proj- 
ects. We use this as a basis for making a range of classroom manage- 
ment decisions; 

• Which lessons are appropriate for which groups of students? 
What skills should be covered? How much and what kinds of 
instruction and practice are needed? 

• How should students be assigned to groups for instruction? For 
practice? How often should groups be changed? 

• Why is a particular student not doing well? What would help? 
How can I challenge a particular student and keep interest 
high? 

• How are my students doing? Compared with last year's group? 
Compared with other classes? Compared with grade-level prog- 
ress criteria? 

• Do my students know what they're supposed to know about this 
unit? Have they acquired the skills/concepts specified in the 
course of study? Do they know it "well enough"? 

• What kinds of lessons/activities will help students who are not 
meeting standards to improve? Who should be involved in 
helping students, parents, aides? 

• How can I do a better job? Why did today's lesson bomb? Hov 
can I provide more useful comments on student papers? Wha 
would make this lesson more relevant to students? What kinds 
of activities would make the lesson concepts easier to under- 
stand? 

It's a simple transition to move from classroom to school-level evalu- 
ation. Substitute the word "program" for "classroom," and "school" for 
"student," and you will have identified some questions that are cen- 
tral to school improvement. 

In a nutshell, regardless of level, much of our routine progress 
tracking is designed to answer two simple questions: 

• How are we doing? 

• How can we improve? 

If we add to these two questions an equally important query, "How can 
we share our successes?" we have captured the purpose of school-based 
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evaluation. We all bring to this school-level evaluation process much 
experience in gathering information that will help us do our jobs better* 



Although school-based evaluation is a powerful management 
technique, a warning to potential users is in order. Evaluation has 
strong potential utility, but its promise has generally exceeded its 
delivery. Evaluation, especially in schools, has too often been identi- 
fied as "testing." Although test data may be useful in tracking your 
progress, to define evaluation narrowly as testing can limit the utility 
of an evaluation. Often imposed from the top down, standardized tests 
are seen by many outside of the schools as the key measure of 
educational quality. Tests have been used to satisfy legislators and 
administrators at the federal, state, and even local levels who wished 
to know how mandated and other special programs were working and 
whether schools were effective. Until recently, school people — teach- 
ers and administrators— have been viewed primarily as data provid- 
ers rather than data users and as implementors of reforms rather 
than initiators of such efforts. 

Limitations of Teachers and local program designers, meanwhile, have raised 
standardized tests, serious questions about the validity of these top-down assessments, 

arguing that required tests often are not well matched to their program 
goals, do not reflect their instructional techniques, and are inappro- 
priate for particular groups of students (Herman, 1990; Herman & 
Dorr-Bremme, 1983). These critics have been joined by prominent 
researchers who also have questioned the value of standardized tests 
for such a broad range of purposes (Eisner, 1985; Sirotnik & Burstein, 
1985). The most widely cited critics of current testing practice assert 
that standardized tests 

• provide a very limited view of educational quality; 

• examine only a narrow slice of the curriculum; 

• emphasize ba sic skills at the expense of higher- order reasoning 
or content learning; and 

• ignore the multiplicity of academic, social, and vocational goals 
that schools are supposed to address. 

Furthermore, there have been serious questions about what standard- 
ized test scores really mean (Cannell, 1987; Linn, Graue, & Sanders, 
1990). Do increased scores denote improved achievement or do they 
reflect a narrowing of the curriculum and teaching to the test? Do 
high test scores indicate that students can indeed read adequately, 
write well, and solve problems involving mathematics in real-life set- 
tings? In short, the utility of current measures of student achieve- 
ment forjudging educational quality may be severely limited. 



What Are the 
Pitfalls in 
Evaluation? 
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Testing is not 
evaluation. 



But, even if better tests were available, more broadly based tests 
administered, and the results more sensitive to local social and commu- 
nity contexts, testing would not be synonymous with evaluation. Test 
scores, or any other single piece of information, provide only a partial 
picture of school quality and are incomplete for the purposes of 
educational planning and school decision making. Tests cannot pro- 
vide full answers to such important questions as: 



Evaluation is not 
limited to 
comparing 
programs or schools. 



Data don't solve 
problems. 



• Why are outcomes as they are? 

• Which parts of the school program are working well? Which are 
not? 

• Which students are benefiting most? Who needs more attention? 

Understanding the processes that create or contribute to school out- 
comes and the contexts in which these relationships occur is a pre- 
requisite for developing a program improvement agenda. 

Just as the reliance on test scores alone provides a narrow view 
of the evaluation process, engaging in evaluation activities only for 
the purpose of deciding which program or school is "better" gives short 
shrift to the wide range of purposes to which evaluation is suited. At 
the school level, where curriculum goals may be decided by an outside 
agency and where decisions about whether to retain or replace a 
program can be dependent upon a state textbook adoption cycle or 
system-wide change, evaluating a program to "see if it's any good" 
often is not an option. Rather, the focus is on "how can we improve 
what we're doing?" Similarly, school-level program decisions center 
on identifying needs, setting school site or program priorities, and 
tracking progress both in implementing change and in student out- 
comes. Although large-scale federal and state evaluations still focus 
on judgments of worth and comparisons of programs, more current 
evaluation theories emphasize the role of data-gathering activities to 
aide decision making and build support for institutions and programs 
(Alkin, 1985; Patton, 1988). 

Those of us old enough to remember Skinner boxes, teaching 
machines, the age of video, and management by objectives have 
learned that a technique or technology cannot transform a school. The 
power of evaluation does not lie solely in its "scientific" methods or 
"hard" data (numbers). We know they're hard because we have to 
crunch then prior to digesting them. Evaluation information alone 
cannot solve problems; only thoughtful and empowered educators will 
solve problems. The power of evaluation rests in its ability to help 
people identify where they are going, how to improve the journey, and 
whether they have arrived. It is a process for communicating, building 
support, and developing a shared vision among the school community. 
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A Better The litany of problems in current evaluation practice contains 

Approach: the roots of a more productive, sensible model, one that better sup- 

School-Based ports restructuring and school-based decision making. What are the 

Evaluation features of sensible, school-based evaluation? 

Sensible evaluation. Sensible evaluation is formulated in collaboration with program 

administrators and implementors. Sensible evaluation is useful; use- 
ful evaluations are designed with input from both those who know 
the program best as well as those who are responsible for improving 
the program. 

Sensible evaluation is wnll aligned with school goals and priori- 
ties. It matches tools and u essments with what the school or pro- 
gram is trying to accomplish. Sensible evaluation focuses on what 
school people agree are important goals. 

Sensible evaluation, while maximally responsive to the needs 
and concerns of the school, is at least minimally responsive to the 
accountability and monitoring concerns of the local district and state. 
Sensible evaluation provides information on general indicators while 
documenting progress on specific local goals. 

Sensible evaluation focuses on the theories guiding program 
actions as well as the short- and long-term targets of change. Educa- 
tional programs are based on theories, often implicit, that guide 
program development. These theories define what actions are re- 
quired to improve education for children, and they predict that, if 
certain activities occur, certain outcomes can be expected. Sensible 
evaluation focuses both on the current theory-based actions as well 
as on the predicted outcomes. For example, one school might be 
interested in exploring the effects on student language acquisition of 
implementing a whole language approach to literacy instruction. The 
ultimate target, of course, is student language performance, a target 
that realistically may not be dramatically affected for several years. 
The short-term view, however, can focus on what needs to happen in 
the meantime: comprehensive staff development, acquisition of new 
materials, practice in implementing new techniques, parent educa- 
tion. Evaluation needs to attend to the critical features of the theory 
of action underlying the whole language program so that progress can 
be monitored over both the short and the long term. 

Sensible evaluation, though guided by theories of action, is at- 
tentive to unanticipated side effects. Although we focus on implement- 
ing certain theory-based strategies hoping to accomplish intended 
goals, we need to be sure that we're not falling behind or causing 
havoc with other school outcomes. If, for example, we are focusing on 
the effects a program is having in raising student achievement, we 
might want to check that these gains are not coming at the expense 
of student attitudes or even attrition. Or, in another example, if a 
small group of teachers is involved in a project, are there negative 
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side effects in their relations with the rest of the faculty? Negative 
side effects indicate areas needing attention. Positive side effects add 
evidence of the success of the program. 

Sensible evaluation designs encompass a variety of indicators of 
school quality and student accomplishments. Multiple measures tri- 
angulate on a variety of important outcomes. Schooling is a complex 
process with complex outcomes; its quality cannot be captured by a 
single score on a single test given on a single occasion. Sensible evalu- 
ations include a variety of measures that assess 

• a range of outcomes; 

• important dimensions of school, program, and instructional 
processes; and 

• key aspects of school context and demographics. 

Sensible evaluation uses measures yielding valid inferences about 
programs, processes, and outcomes. The validity of currently avail- 
able measures for school decision making is only approximate, with 
some current measures more open to question than others. One step 
tow^'d improving our decisions about schools is to use measures 
aligned with the critical features of the local school context, processes, 
and outcomes. A second safeguard against the imperfection of our 
current tests is the use of multiple measures to replicate findings of 
school effects so that we can be more confident about what has 
occurred. 

Sensible evaluation concentrates on doing a few things well using 
as much existing information as possible. Schools are busy places that 
happen to produce a lot of information as by-products of the kind of 
work school people do. Building upon existing information minimizes 
the burden of data collection and maximizes the usefulness of cur- 
rently mandated school records and information archives. 

Sensible evaluators know that certain kinds of evaluation activ- 
ities provide staff both with insights about curriculum and with new 
skills. A prominent example of an evaluation activity that has a large 
staff-development payoff is scoring of performance assessments. In 
the process of reviewing student work and refining or developing 
criteria for rating, staff come to better understand the theories un- 
derlying and predicting student performance. 

Sensible evaluation examines progress over time. Because mea- 
surement in education, as in most fields, is inexact and subject to 
error as well as to influences from a variety of factors, and because 
programs are not implemented under pristine conditions, conclusions 
based on one point in time are unlikely to be reliable. Judgments of 
success about innovations are more credible when the analysis pro- 
vides trends for several years both before and after an innovation is 
implemented. Use of an extended time frame for making decisions 
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about schools is also a response to the reality that change is slow; it 
takes time to get new approaches up and running and producing 
results. 

Sensible evaluation has explanatory power. When an evaluation 
includes information about school and community contexts and on 
processes as well as outcomes, we can examine the interrelationships 
among these to understand why things are as they are. Without an 
understanding of these interrelationships, evaluations cannot really 
answer the question of why a program is working well or what actions 
should be taken to improve it. 

When to Track Because evaluation requires administrator and staff time, the 
Your Progress scarcest commodity in schools, you should carefully select the decision 
With Evaluation areas to be informed by it. Although no general principle exists that 

can tell you: "you need to do an evaluation in this circumstance," the 
questions below will help you identify the decision contexts where 
formal evaluation processes are routinely used: 

• Is this a high-stake decision? Will many students or teachers 
be affected by the outcome of the actions to be taken? 

• Do I need highly credible or even legally defensible information 
to support this decision? 

• Do we need to know how a program works and why it produces 
the effects it does? 

• Is my funding contingent on providing information to describe 
a program or its effectiveness? 

While you are considering whether evaluation is appropriate, 
you should identify potential "stakeholders" for your evaluation — 
those groups expected to act on findings and others likely to be 
affected by or interested in your results. Stakeholder groups may 
include teachers, administrators (site, district, or state), parents, 
students, board members, or specific community constituencies such 
as the business roundtable, civic groups, social service providers, or 
realtors. It is important to involve stakeholder trroups early and often 
in the evaluation process. Their involvement will help to build con- 
sensus, or at least common understanding for negotiation, about 
decisions based on evaluation finding. Involvement of stakeholders 
will promote the credibility of findings and will increase the likelihood 
that findings will indeed influence action. Stakeholder involvement 
not only increases ownership in decisions influenced by the evalua- 
tion but also increases the pool of available help in focusing the 
evaluation, defining its guiding questions, gathering information, 
and understanding results. In other words, consider early who your 
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most important stakeholders are and consider convening their repre- 
sentatives in an ongoing advisory group for your evaluation efforts. 

Your Once you are satisfied that you need an evaluation and you have 
SS in identified relevant stakeholder groups, you are ready to implement 
Steps ^e six-step decision-making process we call "evaluation. " Although 
others have suggested a different number of steps, our model shares 
with many evaluation models a number of common features: 

• a process for clarifying the actual decisions or actions to be 
taken as well as the questions, issues, or goals of the evaluation 
activity; 

• a reliance on multiple sources of information for decision making; 

• an externally verifiable decision-making process; and 

• a belief that data will indeed result in better decisions about 
schools. 

Steps in the evaluation process are neither linear nor fixed, Some 
steps may be taken simultaneously, and there is often an interplay 
between steps, with information from one step changing activities or 
decisions made at another. Like all problem-solving strategies, the 
evaluation process is recursive. An insight gained in one step might 
cause rethinking of the evaluation questions or a revision in data gath- 
ering instruments. A review of results may shift the focus of the eval- 
uation or suggest additional or more important questions. And data 
collection, analysis, and reporting may occur over a very short time 
period so as to appear almost simultaneous. 

In our work at CRESST (the UCLA Center for Research on 
Evaluation, Standards, and Student Testing), we have found the 
following steps to be useful in guiding our evaluation activities: 

(1) Focus the evaluation. Determine the purpose(s) of the evalu- 
ation: the decisions to be made, the possible audiences or peo- 
ple affected by these decisions, and the questions we might 
ask that would enlighten our decisions. 

(2) Identify tracking strategies. Determine what information we 
might need to answer our questions or better understand the 
consequences of our decisions. Make initial decisions about 
the kinds of instruments needed. How will information be 
collected? Will there be interviews? Observations? Focus groups? 
A review of extant data? Student work samples? Standard- 
ized test results? 

(3) Manage instrument development and data collection. Deter- 
mine from whom we need information and when. Make a plan 
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for instrument development if necessary. How long will it 
take? Who will collect information? What will it cost in time 
and resources to purchase or develop and administer instru- 
ments? 

(4) Score and summarize data. Think about the kinds of scores 
you will need to answer your evaluation questions. Choose 
appropriate scores or scoring strategies if you are using pro- 
jects, performance tests, or portfolios. 

(5) Analyze and interpret information. Marshall your score sum- 
maries to inswer your specific evaluation questions. Work 
with stakeholders to "make sense" of findings and conclusions 
in lignt of shared experiences and possible conflicting interests. 

• Look for trends over time to identify strengths and areas 
for improvement. 

• Look for relationships among program processes, student 
and staff characteristics, and outcomes to explain findings. 

• Negotiate a common understanding of findings: make mean- 
ing of the trends, profiles, and summaries of questionnaire, 
test, interview, observation, and/or performance information. 
Find possible courses of action related to findings. 

(6) Act on findings and continue program monitoring. Commu- 
nicate your findings in a timely and appropriate manner. Com- 
mon methods of communicating findings include school action 
or school improvement plans, informal meetings, panel discus- 
sions, formal presentations, and written reports. Report the 
results that are useful and clear to your primary stakeholders 
and other interested audiences. Use different reporting methods 
to match the audience needs. Develop plans to address school 
weaknesses and celebrate strengths. Be sure to monitor your 
plans with — what else? — evaluation. 

Table 1.2 summarizes the steps in school-based evaluation and high- 
lights the principles of sensible practice. 

How to Use Tracking Your School's Success is a "user's manual" for school- 
This Manual based evaluation. It is written as a guide for those new to the evaluation 
process as well as a troubleshooter's manual for those addressing spe- 
cific program planning, implementation, and accountability concerns. 
The following chapters present guidelines for conducting each of the six 
steps and include real-life examples to guide your own evaluation 
efforts. For those wishing to explore the evaluation process more deeply, 
Suggested Readings are provided at the end of each section. 
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TABLE 1.2 Principles of Sensible Evaluation 



Step 



Sensible Actions 



Focus the Evaluation 



Identify Tracking 
Strategies 

Manage Instrument 
Development and 
Data Collection 

Score and 
Summarize Data 



Analyze and Interpret 
Information 



Act on Findings and 
Continue Program 
Monitoring 



Involve significant constituencies 
Include improvement and accountability 
concerns 

Look at long- and short-term targets of change 
Look at relationships between processes and 

outcomes based on your theories of action 
Look for unanticipated side effects 

Use strategies well aligned with school goals 

Use multiple indicators 

Build in checks for validity of inferences 

Build measures upon existing information 
Consider accountability mandates 
Match instruments with specific evaluation 
questions 

When appropriate, use scoring sessions for 
staff development 

Assure valid inferences by choosing appro- 
priate scores 

Involve key constituencies 
Examine progress over time 
Consider and refine your theories of action 
r ie alert to unanticipated side effects 
Corroborate findings by using multiple 
indicators 

Use the principles guiding focusing activities 
to monitor actions based on your findings 



Worksheets are provided in Resource D at the end of the book; 
they have been designed to be used as templates for your own evalu- 
ation efforts. You should feel free to repro luce the templates and to 
modify them to fit your particular situation. Sample versions of the 
worksheets with possible answers filled in appear *n the text. 
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Suggested For those wishing more information about evaluation in general 
Readings or the overall procedures, we offer the following suggestions: 

Berk, R. A., & Rossi, P. H. (1990). Thinking about program evaluation. 
Newbury Park, CA: Sage. 

Berk introduces the range of questions answered by evaluation 
and describes methods currently used. Especially helpful are the 
specific examples of evaluation goals and methods. 

Herman, J. L., Morris, L. L., & Fitz-Gibbon, C. T. (1987). Evaluator's 
handbook. Newbury Park, CA: Sage. 

The Handbook provides step-by-step guidelines for every phase 
of evaluation from selecting questions to preparing the report. A 
valuable resource for getting an overview of the nuts and bolts 
in planning and managing an evaluation. Of particular use is 
first chapter's synopsis of major evaluation models. 

Popham, W. J. (1988). Educational evaluation (2nd ed.). Englewood 
Cliffs, NJ: Prentice-Hall. 

Popham's provocative and humorous prose guides the practitioner 
through evaluation models, writing objectives, measuring achieve- 
ment and attitudes, evaluation designs, sampling strategies, data 
analysis, and reporting. It is less specific than the Handbook but 
an almost painless way to learn about the more technical aspects 
of evaluation. 

Rutman, L. (1980). Planning useful evaluations. Beverly Hills, CA: Sage. 
Do you need to know if your program is evaluable? Before tack- 
ling the Augean stables, you may wish to read Rutman's descrip- 
tion of factors that affect the evaluability of programs or use his 
program analysis tools in reviewing your school. Suggestions for 
enhancing the evaluability of a program are especially helpful. 
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Step One 

Focus the Evaluation 



During this first step, you will focus your evaluation by formu- 
lating the specific questions you want to address. These identify 
the specific areas of your school program that will be exam- 
ined. Plan to convene your advisory board — or other mecha- 
nism for involving stakeholders — to assure that the interests of 
important constituencies are represented. 



In the context of school-based management and restructurings a pri- 
mary purpose of evaluation is to monitor and improve the quality of 
school programs or the components of those programs. A second purpose 
is to provide at least modestly credible data about school progress to 
district personnel and the public. The simple question, "How are we 
doing?" perhaps best expresses the focus of school-based evaluation. 
Deceptively simple and straightforward, yet related to nearly all school 
functions, the "how are we doing* question subsumes a host of school 
staff concerns: What student, community, or curricular needs should we 
be addressing? What should we change in existing programs or what 
new programs should we adopt to meet these needs? Are we doing well 
enough? How can we do better? Although "how are we doing?" suggests 
a concern with "how can we do better?" the question is not limited to 
examining areas for improvement. Too often, we think of evaluation as 
an activity to identify school weaknesses when, in fact, it provides ample 
opportunities for documenting what we do well. All schools have success 
stories to share and strengths upon which to build. 

17 
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FOCUS on Major Yet, simply asking, "How are we doing?" is not enough to provide 
Program Aspects a clear evaluation focus. What we should be examining depends upon 

political and legal requirements, our educational philosophy, and our 
values. For example, one way to focus our evaluation is to attend to, 
among other thingc what our state or district insists be examined. 
School accountability report cards, required in a number of states, are 
one instance of a mandated focus. These required areas of reporting 
can provide a starting point *br thinking about important dimensions 
of school quality. When customized to a school's unique goals and 
circumstances, such external mandates enable us to highlight our 
school successes as well to monitor improvement efforts. Furthermore, 
whether a school improvement effort is sparked by an outside man- 
date or local initiative, certain aspects of schooling appear time and 
again as foci of school quality: 

• Student outcomes: achievement, attitudes, school completion, 
preparation for work or college 

• Curriculum: adequacy and effectiveness of educational materi- 
als, technology, resources; content coverage; course work re- 
quirements; access by different student populations; alignment 
with state/school goals and assessments 

• Instruction: strategies, grouping, teacher role, classroom orga- 
nization, student interaction, use of technology 

• School climate: consensus on school goals, expectations for 
student achievement, relationships among and between stu- 
dents and teachers, pride in and satisfaction with school, ad- 
ministrative leadership, teacher empowerment, support for 
innovation, safety, student and faculty morale 

• Staffing: staff preparation, professional development, student- 
staff ratios, staff understanding of student population needs, 
staff demonstration of special competencies such is being bi- 
lingual, understanding mainstreaming techniques or curricu- 
lum requirements 

• Parent-community relations: attendance at school events, par- 
ent involvement with child's educational progress, understand- 
ing of school goals, support for and value of school, volunteer 
programs, financial support, business partnerships 



Develop a 
Comprehensive 
View of School 
or Program 
Quality 



These aspects of schools are of interest, of course, because many 
people believe them essential to quality schooling. Although much of 
public interest centers on student outcomes, those of us in schools 
know that improvements in student outcomes cannot be accomplished 
without changes in these other areas. When we look for reasons that 
outcomes are as they are, as mentioned earlier, we clearly need to 
look at other aspects of the school environment. 
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Figure 1.1. A Process Outcome Model of Schooling 

And, if we want evaluation to help us to understand how our 
programs are operating and to clarify underlying reasons for current 
outcomes, it's helpful to have in mind a model of how these various 
aspects fit together. Figure 1.1 maps one possible set of relationships 
between program features and student outcomes. Here we see that 
student outcomes are thought to be influenced by school and instruc- 
tional processes oi climate, curriculum, and instructional strategies. 
These in turn are influenced by teacher background and training, 
parent involvement, and other student and school factors. If student 
outcomes don't match our goals or expectations, then we need to 
recycle back through these other areas to identify potential causes 
and design remedies. Although your "model" may have different 
features and suppose different sets of relationships, the point is that 
it is useful to have a broad view of how different aspects of a school 
fit together. 

Each of the areas displayed in Figure 1. 1 defines potential areas 
of focus for an evaluation. At the most general level, we can ask, "How 
are we doing in student outcomes? In curriculum? In instruction? In 
school climate?" This is a preliminary step in the focusing process. 



Choose a 
Wide-Angle or 
Close-Up 
Perspective 



As you identify program area(s) of interest, consider whether you 
need a broad-spectrum or close-up view of your program. When you 
are writing a school plan for the first time or deciding upon what 
might make your school a place that better meets student needs, you 
will want to look at the big picture. Needs assessment can help give 
this perspective; it provides the wide-angle shot across the entire 
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school. Like many school report cards, a needs assessment examines 
the overall quality of school programs and seeks to identify areas of 
strength and weakness. 

In contrast, when you have a specific program you are trying to 
implement, rather than looking across the whole school, you may 
want to know how well that program is working and how to improve 
it. In this instance, you will engage in the process known as formative 
evaluation, looking in greater depth at the operation and effective- 
ness of specific change efforts. 

Both perspectives, either wide angle or close up, address the 
same general questions: "How are we doing?" "How successful are 
we?" In needs assessment, we focus broadly on the outcomes of what's 
happening in our school. In formative evaluation, our emphasis is on 
how and why it is happening. 

An initial task in focusing the evaluation, then, is to decide 
whether our interests are best served by needs assessment or by 
formative evaluation. We then must narrow our interests to a man- 
ageable number of concerns. 

Needs As mentioned before, needs assessments are often required steps 
Assessment in program planning and in the school goal setting process. In con- 
Questions ducting a needs assessment, we look broadly across all curriculum 
areas. Having identified an area or two in which we are doing well, 
we may highlight this with our public or choose to share our successes 
with other professionals. Certainly, identification of areas of strength 
is central to the accreditation and school review process. At the same 
time, we may find an area or two where we have a need to improve. 
The needs assessment process enables us to look more deeply at these 
areas in a diagnostic sense and to establish some priorities for pro- 
gram planning or revision. 

The needs in needs assessment most often refers to student needs 
based on current or past performance data such as standardized test 
scores, course enrollments, post-high school plans, or language profi- 
ciency. State and federal project funding for such projects as Title VII, 
Chapter 1, or state innovative program grants requires that schools 
justify their need for a particular project by presenting student outcome 
data pointing to deficiencies the project will correct. Parent and staff 
perceptions, often garnered through beginning or year-end surveys, 
also are often taken into account. 

Although, typically, these mandated project-related needs as- 
sessments focus on basic skills achievement data (i.e., reading and 
mathematics standardized test scores), you need not be so limited. 
You will want to assess other areas as well, either because they are 
important problem areas in your school or because they represent a 
priority for you and/or your important stakeholder groups. Your school 
might wish to look, for example, at needs in such areas as school 
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climate, parent involvement, instructional quality, use of technology, 
or school image/public relations. 

An important aspect of needs assessment is the identification of 
the group(s) having the need. If you only examine results for the 
school as a whole, you may believe that you are "on track." Unfortu- 
nately, these overall results may mask significant problems in spe- 
cific subgroups of students, such as by masking serious performance 
disparities. Which subgroups might be of interest? You might first 
look at students participating in special programs, special education, 
Chapter 1, and bilingual or gifted education. You might pose ques- 
tions about groups that past experience or research have shown to 
have special needs. Among these subgroups are students new to a 
school, working students, girls (in mathematics and science), boys (in 
writing), or students with erratic attendance. By asking how particular 
subgroups may be doing, you attend to important equity concerns. 

The strategy of asking about specific results for different sub- 
groups also applies to areas other than student outcomes. Different 
subgroups of parents, teachers, or aides and volunteers might have 
different insights about the program that could be obscured when 
data are averaged across the entire population of responses. If you 
have some notion that different groups might have very different 
opinions on school operations and/or impact, you should keep these 
groups in mind. For example, there may be a difference between 
neighborhood parents and parents of students bussed into a school in 
their perceptions of how well the new student self-esteem program is 
working. Or social studies teachers may or may not be observing 
effects of the new literature-based writing program implemented in 
the English classes. 

Table 1.1 illustrates typical needs assessment questions and may 
provide a starting point for your own needs assessment efforts. 

Formative Formative evaluation questions emerge logically from needs as- 
Evaluation sessment questions. If a group is not performing well in some area or 
Questions parents are concerned about a program, we institute some change to 
address the need. As we track our progress in implementing this 
change, the needs assessment questions that address program as- 
pects remain useful. In particular, we will want to continue to track 
the needs our change(s) were designed to address. 

The path we follow in formative evaluation, however, differs some- 
what from that taken for a needs assessment. Although needs assess- 
ment questions generally ask us to look at outcomes of one kind or 
another for specific groups, formative evaluation allows us to look at 
processes as well. In formative evaluation, we can augment the "how 
are we doing" question with the more interesting query, "Here's what 
we're trying to do, what needs to happen to make that a reality?" Most 
often, what we are trying to do is to make a curricular, organizational, 
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TABLE 1.1 Typical Needs Assessment Questions 



Student 
Outcomes 



Instruction 



After actfusting for changes in test norms or student mo- 
bility, in which areas tested by standardized tests have 
we improved over the last 3 years? Maintained our rank- 
ing? Declined? 

Are there performance differences by language profi- 
ciency, sex, or ethnicity? 

What are student attitudes toward history, science, 
math, and/or English? Do attitudes differ by grade 
level? By particular subgroups of students? 

What is our school completion rate? How does it com- 
pare with comparable schools? State averages? 

Curriculum Do we have enough core literature books for each child 
to take books home? 

Is our curriculum aligned with state frameworks or na- 
tional curriculum standards? Do we have clear goal 
statements for each curriculum area? 

Are course goals and expectations comparable for all stu- 
dents regardless of ability or language proficiency? 

How does our classroom instruction differ from what we 
consider desirable or optimal practice? 

How is technology used to support instruction and stu- 
dent learning at each grade level? 

What kinds of in-service does staff suggest? 

Do parents and students perceive staff as holding high 
expectations, being involved with students, and being 
well prepared? 

Do parents feel comfortable visiting the school? Discuss- 
ing their child's progress with teachers? Discussing 
school concerns with administrators? 

Do we provide volunteer opportunities for parents re- 
gardless of their educational background, language, or 
work obligations? 

What community and business partnerships are support- 
ing specific programs and where do we need more support? 

Is there mutual respect and a positive working relation- 
ship among students? Between students and teachers? 
Between teachers and parents? Between teachers and 
administrators? 

Do students take pride in their school? 

Do teachers feel supported by administration? 

Is there suppport for change or innovation in this school 
when such a need exists? 

Do students and teachers feel physically safe at school? 



Staff 



Parents and 
Community 



School Climate 
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or instructional change that we hope in the long run will improve 
student outcomes. In the short run, the changes may be aimed at 
improving the school climate, increasing parent involvement, or chang- 
ing teaching practices. 

Frame your Theories of action are the rationale for why we are using partic- 
questions in theory ular programmatic strategies. Theories of action define what essen- 
of action, tial program features are, how they fit together, and why certain 
program features are thought to be beneficial for students. For exam- 
ple, if you are trying to change your primary grade curriculum to a 
whole language approach, your theory of action tells you what the 
important elements are in that approach and what needs to happen 
for teachers to change to such an approach. Among the features you 
would expect to see in a whole language classroom are teachers modeling 
the reading process by reading aloud to students; students using 
"rear literature as opposed to basal readers; students engaged in a 
variety of reading activities from sentence strips to pattern books to 
student published material; students being asked to reconstruct whole 
stories, make predictions, identify familiar words; and students writ- 
ing or talking about their reading. Among the things that might be 
necessary to encourage teachers to change to such an approach are 
quality training, follow-up, and availability of appropriate materials. 

As you form your evaluation questions, you will want to focus on 
whether in fact "whole language" is occurring and on whether neces- 
sary precursors occurred as well as on the actual reading progress of 
the students. When you look at programs as models of instruction and 
include questions about processes as well as outcomes, your evalua- 
tion results will point out specific ways to improve the program. 

By linking your evaluation questions to a larger view of what 
should be happening in school, you will be able to gather information 
more directly useful for program improvement. Let's take an example 
that might occur in any school. During the first year of whole lan- 
guage implementation, student standardized reading test scores drop 
in some classes but not in others. You find that, because of the way 
students are assigned, the drop is not attributable to student charac- 
teristics. Because you have a theory of action about what whole 
language instruction is supposed to look like (and produce), you also 
visited classrooms and noted which aspects teachers were using. In 
reviewing your observation notes, you find that, in the classes where 
scores declined, teachers were using more whole language strategies 
than i i ;lassroorrs where scores went up. There were no drill and 
practice or testlike activities. In fact, the only time students encoun- 
tered a multiple choice format was on the district standardized test. 
Your theory of action, however, prepared you for this eventuality, so 
you collected some information related specifically to expected out- 
comes from a whole language program. Because your action theory 
suggests that students will be independent readers, you inserted a 
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few questions into your annual parent survey to find out what stu- 
dents read at home and how often they volunteered to read aloud to 
their parents. You also asked the parents to rate their child's attitude 
toward reading. Sure enough, parents of students in classrooms where 
more whole language strategies were used (and where test scores 
dropped) reported more positive student attitude toward reading and 
more at-home reading. These data now provide some suggestions for 
program improvement. You want to maintain positive attitudes yet 
prevent test scores from dropping drastically. Thus you may wish to 
have those "further-along* teachers provide at least some practice in 
test taking while coaching the more traditional teachers in whole 
language instruction. 

Table 1.2 summarizes a few current trends in education. Many of 
the changes we are implementing in our schools, from whole language 
instruction, to process-based writing, to meaning-based mathematics, 
incorporate one or more of these trends. The list may suggest to you 
what action theory(ies) are related to your own programs and what 
kinds of questions about program processes you could ask. 

Consider the As we are well aware, school improvement is a complex process. 

factors underlying The literature on innovation and change in schools suggests that sue™ 

successful cessful change depends upon several concomitant factors (Herman & 

innovation. McLaughlin, 1978): 

Teacher consensus about the need for and ownership in the change 
Leadership support of the effort 

Adequate staff development and follow-up assistance on the im- 
plementation specifics 
Teacher/staff/school adaptation of the change to meet local needs 
Materials and resources required to implement the change 

The^e precursors to successful innovation or change provide areas 
about which we might ask evaluation questions. They also suggest 
possible reasons why your programs are or are not working well. 

A well-known evaluation model, the Concerns Based Adoption 
Model (CBAM), was developed specifically to answer evaluation ques- 
tions related to school-based innovations. Those who wish an in-depth 
view of this specialized kind of formative evaluation should read the 
well-written ASCD publication Taking Charge of Change (Hord et al., 
1987). 

Table 1.3 lists typical formative evaluation questions to spark 
your thinking about your own programs. 
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TABLE 1.2 Current Trends in Education 

Active Learning 

The emphasis in both instruction and assessment has shifted from get- 
ting students to respond to having them produce or demonstrate what 
they know. Meaningful learning is now thought to occur when students 
have opportunities to tackle real discipline-based problems and inter- 
act with the "tools" of the subject. In mathematics, students manipu- 
late concrete objects and derive concepts inductively; they graph, they 
weigh, they measure, they draw, they write, and they think aloud 
about real-world problems and how to solve them mathematically 
using their own strategies. In language arts or literature, students 
have some choice in what they read and are asked to add their own in- 
sights to well-established interpretations. In science, students experi- 
ment and manipulate materials instead of reading about experiments 
or watching demonstrations. Sometimes the experiments are computer 
simulations, sometimes actual experiments. 

Collaborative/Cooperative Learning 

In collaborative/cooperative learning, students work in groups on com- 
mon tasks. Collaboration and cooperation have different manifesta- 
tions. Most versions advocate heterogeneous grouping. Some versions 
require that students be graded individually; others require that they 
receive a group grade. 

Contextualized Learning 

The impetus for contextualized learning comes from recent research in 
cognitive psychology, which suggests that "problem solving" or "deci- 
sion making" or even "reading" have different meanings in different 
disciplines or contexts. The natural extension of this hypothesis is that 
skills need to be taught in context. Reading, for example, is really 
reading about science, literature, social studies, art, mathematics, and 
so on. Students are asked to apply skills in a variety of contexts under 
the assumption that the discourse demands change when the context 
changes. This view calls into question units or courses that emphasize 
skills apart from subject matter. 

Developmentally Appropriate Instruction 

Many programs, especially for elementary children, are now founded 
on developmental learning principles, summarized in Bredenkamp 
(1987). Developmentally appropriate teaching strategies include an in- 
tegrated curriculum so that children's learning in all traditional sub- 
jects occurs primarily through projects and learning centers that re- 
flect the children's interests and suggestions. The teachers guide 
children by extending their ideas, responding to their questions, and 
challenging their thinking. Children are encouraged to evaluate their 
own work and to understand that errors are a natural and necessary 
part of learning. Teachers analyze children's errors to plan curriculum 
and instruction. 
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TABLE 1.2 Continued 

Integrated/Interdisciplinary Instruction 
Young children have difficulty separating their world into "subject do- 
mains." The separation of learning into "disciplines" is often artificial 
and ignores the interconnectedness of knowledge. It is now more com- 
mon to see the elementpry curriculum organized around themes or in- 
terdisciplinary units that integrate the subject matter of science, social 
studies, art, and/or music with the skills developed in reading, lan- 
guage arts, and mathematics. At the secondary level, interdisciplinary 
humanities and math-science programs are becoming more common. 

Prior Knowledge as a Basis for Learning 

Learning is more efficient and meaningful when students can use what 
they already know to make connections with new material. Prewriting 
and prereading activities, wh^re students are asked to recall or brain- 
storm what they know related to a particular topic, are strategies de- 
signed to help students build upon what they know and to refine exist- 
ing concepts. We recognize that we rarely "start from scratch." Spend- 
ing time to help students discover or organize what they know related 
to new learning helps them make connections between prior and new 
knowledge and helps teachers to plan instruction based on student 
needs. 

Thinking Skills as an Instructional Outcome 
Although thinking skills are to be taught in context, transferrable 
skills, rather than information, are desired outcomes of instruction. 
The California Frameworks, for example, ask that students develop a 
range of thinking abilities beyond recall and comprehension. These 
thinking skills include critical thinking, problem solving, science pro- 
cess knowledge, aesthetic appreciation, and meaning-based commun- 
ication. The frameworks have as a goal the development of "generic* 
thinking techniques such as analyzing from multiple perspectives, 
drawing inferences, and evaluating outcomes. 

Authentic Performance Assessment 

A trend in classroom testing has been to design tests that look much 
more like reaMife tasks. There is a move to add performance assess- 
ments to paper-pencil tests as a way to evaluate student learning. 
Meaningful assessment is viewed as an integral part of instruction and 
not separate from it. Meaningful assessment asks students to 
demonstrate, produce, reflect, and self-evaluate. Assessment is now 
concerned with diagnosis as well as judgment of progress and focuses 
on process as well as product. Teachers can use such techniques as 
reading/writing logs, journals, tape recordings, running records, and 
other observational methods to supplement traditional teacher- or 
publisher-made end of unit tests. 
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TABLE 1.3 Typical Formative Evaluation Questions 



Student 
Outcomes 



Curriculum 



Instruction 



Staff 



Parents and 
Community 

School Climate 



How has the addition of a computerized math explora- 
tions laboratory affected student attitudes and out- 
comes in required math courses? 

What changes in student motivation and perceptions of 
ability are occurring in our ungraded primary class- 
rooms? 

What changes could we make in the homeroom guidance 
program to improve school attendance? 

What required literature books should be retained next 
year? Dropped? Made optional? 

How can curriculum goals and expectations for student 
performance better meet student needs? 

What changes do we need to make to provide challeng- 
ing problem-solving experiences for all students in all 
subjects? 

What enrichment can be added to the mathematics pro- 
gram to prepare students enrolled in grade-8 math for 
algebra? 

How can we provide instruction in the content areas to 
limited-English-proficient students? 

What strategies have worked best to help low-achieving 
students keep up in heterogeneously grouped classes? 

What kinds of problems has staff had in implementing 
portfolio assessment? In translating portfolios to grades? 

Which staff members can be used to provide coaching 
for others in the use of math manipulatives and mean- 
ing-based math activities? 

How can parent volunteers be used more effectively? 

What changes do we need to make in advisory council 
procedures to get more parent participation? 

How can we improve school spirit and extracurricular 
participation? 

What parts of the dropout prevention program are most 
directly linked to students staying in school? 



Select and You may find you are interested in many more questions than 
Refine Priority you have the time, energy, or wherewithal to address. During the 
Questions focusing stage of the evaluation, however, you should allow yourself 
to be as inclusive as possible. Allow your advisory group to brainstorm 
a range of questions about each aspect of program quality in which 
you are interested (student outcomes, curriculum, instruction, cli- 
mate, and so on). 
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You may wish to use Worksheet 1.1, "Evaluation Focus a id 
Priority Questions" (see Resource D for the blank version; a filled-in 
example appears in Sample Worksheet 1.1) to provide a structure for 
this brainstorming session. Write down one to three (at most) general 
questions you would like to answer in a program area. Then be as 
specific as you can about what you mean by each of these questions. 
For example, simply asking a How are we doing in student outcomes?" 
is too vague. Even asking "How are students achieving in reading?" 
is too vague. Try to specify what you mean by student success in 
reading — standardized test performance, oral reading, motivation, 
reading for pleasure? Similarly, don't simply pose a question such as 
"How good is our school climate?" Instead, pose a series of questions 
that indicate what's important to you in school climate — relations 
among students, between teachers and students, among teachers, 
achievement expectations, and the like. 

Or, if you are oriented toward formative evaluation, trying to 
track the effects of programmatic changes or of innovative programs, 
formulate specific evaluation questions that apply directly to how and 
what your program is trying to accomplish. For example, you would 
not simply want to pose the general question: "Is our dropout program 
accomplishing intended student outcomes?" but would want to ask 
questions about the specific outcomes the program was intended to 
influence— for example, are students remaining in school? Is their 
attendance improving? Are their grades improving? Similarly, while 
the extent of program implementation is a critical issue, you would 
not want to leave your question at this level of generality. Instead, 
you would want to ask about whether critical program features are 
being implemented as intended — for instance, are counselors meeting 
regularly with identified students? Are students attending after- 
school tutorials? Are parents actively involved in their children's 
program? Have identified community agencies been actively involved 
in the program? 

Although articulating specific questions of interest is an arduous 
and time-consuming process, it is time well spent. It will immeasurably 
help you in selecting appropriate progress measures that are sensitive 
to your school's goals. It also will help you to communicate to others 
exactly what you are trying to accomplish, helping to focus not only your 
assessment efforts but your programmatic efforts as well. Furthermore, 
posing specific questions will encourage you to be realistic about your 
program expectations and about posing reasonable evaluation ques- 
tions — questions that are manageable and feasible to answer. 

After you have generated your list, put a check next to all 
mandatory questions: those you must answer to fulfill district or state 
requirements or to meet the goals set in your school plan. Next, decide 
the priority rankings for the other questions. 

How do you decide which questions are most important or should 
be given highest priority? No answer fits all situations. Michael 
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Sample Worksheet 1.1 

Evaluation Focus and Priority Questions 




Priority Rating 


Questions 


Low 


Medium 


High 


Required 


Outcomes: 
How does our dropout rate compare 

with that of the state? 
Which students are "at risk* in 

terms of high school credits, 

attendance, and GPA? 
What is student attitude toward 

school completion? 




/ 


/ 


/ 


Curriculum: 

Are "at-risk students" enrolled 
in our study skills program? 

What materials are being used 
with at-risk students in hetero- 
geneously grouped English and 
social studies classes? 


/ 




/ 




Instruction: 

What special strategies do teachers 
have for helping at-risk students 
within regular classrooms? 

To what extent are teachers 
implementing the "Teacher 
Expectation-Student Success 
Program" strategies? 




/ 


/ 

i 




Staff: 

What does the staff identify as 
causing our dropout rate? 




1 

i ' 




Parents and Community: 

In what ways can parents be involved 
in reducing our dropout rate? 




! 

! 1 / 



Patton (1988) suggests that the screening criteria for any evaluation 
question should be these: "What do I want to know that I don't already 
know?* and "What will it enable me to do that I can't do already?* We 
have found the most useful questions about outcomes are those that 
focus on outcomes highly valued by particular stakeholder groups. As 
for questions related to school climate, curriculum, instruction, staff, 
and parents/community, select those examining processes that are 

• most essential to program success or school effectiveness, 

• most complex/difficult to accomplish, 
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• most costly or resource intensive, 

• most controversial, and/or 

• related to more than one valued outcome. 

Try to limit your "high-priori ty n designation to three or four 
questions, at most, in the program area of interest. That way, if you 
have been overambitious, you can adjust youi focus, Remember, too, 
that you will be revising your questions and making them clearer as 
you progress in your evaluation- The instruments you choose and the 
settings in which you use them could cause you to rethink or reword 
your guiding questions. 

Once you have a set of manageable mandatory or high-priority 
questions, you are ready to consider how you will answer them — our 
task in Step Two. 

References Berman, P., & McLaughlin, M. W. (1978). Federal programs support- 
ing educational change: Implementing and sustaining innova- 
tions (Vol. 3). Santa Monica, CA: Rand Corporation. 

Bredenkamp, S. (Ed.). (1987). Developmentally appropriate practice 
in early childhood programs serving children from birth through 
age 8. Washington, DC: National Association for the Education 
of Young Children. 

Hord, S. M., Rutherford, W. L., Huling-Austin, L., & Hall, G. E. 
(1987). Taking charge of change. Alexandria, VA: Association for 
Supervision and Curriculum Development. 

Patton, M. Q. (1988). Utilization-focused evaluation (2nd ed.). New- 
bury Park, CA: Sage 

Suggested Our suggestions in this section include two kinds of resources: 
Readings Those designed to help you ask useful questions and narrow the scope 
of your evaluation, and those describing action theories underlying 
the curriculum. Knowledge of these action theories should precede 
formative evaluation activities. If you wonder at the relative length 
of this list, it reflects the relative amount of time needed in focusing 
activities. 

Brandt, Ronald S. (Ed.). (1981). Applied strategies for curriculum 
evaluation. Alexandria, VA: Association for Supervision and Cur- 
riculum Development. 

Although this book only focuses on one aspect of school-level eval- 
uation—the curriculum — it provides a variety of examples from 
practitioners that can serve as models for particular school settings. 
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Bredenkamp, S. (Ed.). (1987). Developmentally appropriate practice 
in early childhood programs serving children from birth through 
age 8. Washington, DC: National Association for the Education 
of Young Children. 

A staple in the library of early childhood or elementary education, 
this book provides the theoretical foundations for a developmental, 
child-centered curriculum. The specific developmentally appropri- 
ate practices listed can guide program development and review, 

California Curriculum Frameworks. Sacramento, CA: Department of 
Education. 

So you don't live in California and never would. Nevertheless, 
the state's Department of Eduction has produced a series of 
curriculum frameworks for each curriculum area from art to 
zoology. These frameworks incorporate the latest research in 
student learning, curriculum organization, and assessment. The 
guidelines define "good" programs for today's students. 

Davis, R. B., Maher, C. A., & Noddings, M. (Eds.). (1990). Construe- 
tivist views on the teaching and learning of mathematics. Reston, 
VA: National Council of Teachers of Mathematics (NCTM). 
For those who simply must know more, this monograph in the 
Journal for Research in Mathematics Education introduces the 
theory behind "hands-on" or meaning-based mathematics in- 
struction, explores mathematical thinking, and describes how 
students engage in mathematics. It provides suggestions for 
teachers as well— the real theory behind current mathematics 
curriculum theory. 

Morris, L. L., & Fitz-Gibbon, C. T. (1978). How to deal with goals and 
objectives. Beverly Hills, CA: Sage. 

A step-by-step approach to writing and using goals and objectives 
in the evaluation process. For groups unused to thinking about 
measurable or observable progress indicators, this book is a boon. 

National Council of Teachers of Mathematics. (1989). Curriculum 
and evaluation standards for school mathematics. Reston, VA: 
NCTM. 

An essential document for reviewing mathematics programs, it 
has served as a model for many state mathematics frameworks 
and is a resource for the development of national mathematics 
tests as part of the President's "America 2000" agenda. 

National Council of Teachers of Mathematics. (1991). Professional 
standards for teaching mathematics. Reston, VA: NCTM. 
This is the companion volume to the NCTM Curriculum Stan- 
dards* It is useful in program development md for looking at 
your own instructional practices, and it can serve as a guideline 
for professional development programs. 
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Routman, R. (1991). Invitations: Changing as teachers and learners 
K-12. Portsmouth, NH: Heinemann Educational. 
This is the quintessential practitioner's guide to whole language 
instruction written by a public school language arts teacher. It 
presents whole language theory and teaching and evaluation 
strategies. The "evaluation" chapter is essential, describing what 
happens to standardized test scores (for those who don't already 
know) when whole language programs constitute the curriculum. 
This chapter provides specific alternative assessment and eval- 
uation strategies for K-12 whole language/English/humanities 
programs. 

Stecher, B. M., & Davis, W. A. (1987). How to focus an evaluation. 
Newbury Park, CA: Sage. 

Tailor-made for this first step, this volume of the Program Eval- 
uation Kit provides administrators working with advisory groups 
with the background needed to guide the process of selecting 
evaluation questions. 

Stufflebeam, D. L., McCormick, C. H., Brinkerhoff, R. O., & Nelson, 
C. O. (1985). Conducting educational needs assessment. Boston, 
MA: KJuwer Nijhoff. 

From one of the pioneers in educational evaluation comes advice 
for educators struggling with identifying information needs and 
important constituencies for school evaluation. 




Step Two 

Identify Tracking Strategies 



Overview 



What Evidence 
Are We 
Looking For? 



In Step One, you selected areas of interest, such as student 
outcomes and school climate, as a focus for your school im- 
provement effjrts. You also identified a couple of specific eval- 
uation questions in each of these areas. The task in this second 
step is to find appropriate strategies to help you answer these 
questions. Continue working with your small group of key 
personnel, your advisory council, department heads, or the 
like, to address three questions that are central to identifying 
tracking strategies: 

• What evidence are we looking for? 

• Where and how are we going to find it? 

• What standards will we use to judge our success? 



A first challenge in identifying appropriate tracking strategies is 
pinpointing the specific qualities or changes for which you are look- 
ing. What are the behavioral manifestations of the qualities in which 
we're interested? What are the critical things we can observe to know 
whether desired changes are occurring? What are the specifics of 
what we want to know? Although these perhaps seem like simple and 
straightforward issues, you may find they are neither. Like other 
steps in the evaluation process, you may find instead that the process 
of addressing these issues raises some interesting discussions about 

33 
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what are truly key and meaningful aspects of your school program. 
You will need to think hard and creatively about what is essential 
information for answering each of your evaluation questions. 

The starting point, again, is your evaluation questions. Do they 
suggest the kinds of evidence you will need? For example, if you are 
looking at your dropout prevention program and have asked whether 
counselors are effectively meeting with students on a regular basis, 
you might want to look at the number of appointments counselors 
have had with your target group since the beginning of school and at 
whether the meetings are perceived as effective. Getting more pre- 
cise, you also would want to consider the specific qualities that should 
count as effective— such as the nature of the interactions between 
students and counselors and the content of the sessions. Further- 
more, if you asked whether attendance was improving for these 
at-risk students, you will obviously need attendance data. If you want 
to know whether students' attitudes toward school are improving, you 
might look at attendance, homework completion, or classroom behav- 
ior or whether students say they "like" school. 

To take another example, suppose your interests center on a 
program aimed at cultural sensitivity and integration and that one 
of your questions inquires about whether race relations are improv- 
ing. You might look at whom students choose to eat lunch with, whom 
they socialize with during and after school, whom they choose to work 
with in class when given the opportunity, or whom they say they 
would like to study with, join a club with, or some other question 
related to affiliation. 

What you are trying to define are "indicators" that will give you 
meaningful data to answer your questions. Although there certainly 
is no single correct set of such indicators, Table 2.1 offers a range that 
may be appropriate for answering some of yours. Note that the table 
concentrates mainly on indicators of student outcomes; your efforts 
will want to consider other aspects as well. 



As Table 2.1 demonstrates, what we want to know is often closely 
tied to the specifics of how we will collect our information. For 
example, when we are interested in attendance data, the natural source 
is student attendance records. If we are concerned with perceptions 
about different aspects of our program, asking relevant constituencies 
about their perceptions is a natural option, either through an interview 
or a questionnaire. As these examples show, in deciding where to get 
our evidence, we have two major choices: to use existing sources of 
information or to develop the means to gather new information. 

What's already Concerns for time and resources often dictate that we first ask: 
available? "What information do we already have that is related to our evaluation 
questions?" Do we have records (or could we fairly easily devise ways 



Where and How 
Are We Going to 
Find Our 
Evidence? 
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TABLE 2.1 Sample Progress Tracking Indicators 

Descriptive Indicators At School At District 

Student ethnicity / 

Student language proficiency / 

Student education code: gifted, special education / 

AFDC count / 

Parent occupation / 

School capacity and facility information / 

Teacher credentialing: majors, minors, status / 

Specialist/resource staff / 

Program Characteristics 

Number enrolled in gifted, special education, 
bilingual programs / 

Course objectives and required texts / / 

Enrollment patterns: requirements, electives / 

Grading patterns: grade distributions by 
class/teacher / 

Recent program changes: goals/text/strategies / 

School Climate I Student Attitudes 

Course evaluations / 
Principal observations of teachers being 

evaluated this year / 
Parent comments and phone calls / 
Program review report or state accreditation report / 

Student Outcomes 

Droput rate / 
Average daily attendance / 
Post-high school placement: college, military, work / 
Standardized test scores: state, national / 
Percentage receiving "D," "F, w or "Unsatisfactory** / 
Retention rate (including 2-year kindergaitners) / 
Course-related tests: advanced placement, 

Regents, teacher tests / 
Work-experience placement: kinds of jobs, 

retention, evaluations / 



to obtain them) of attendance, club membership, counseling appoint- 
ments? Do we have computerised summaries of grade point averages, 
a "D/F/Unsatisfactory" list of students v/ho are getting poor marks? 
Where have we filed our reports from the ACT or SAT testing pro- 
grams? Does our state testing program provide information related 
to the performance of "at-risk" students, of limited English students, 
of students new to the school? Do we have records on library check-outs, 
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on college acceptances, on attendance and ratings of teacher in-service 
days, and so on? As we've all no doubt noticed, the record keeping and 
bureaucratic requirements in schools these days produce lots of re- 
cords, some of which may be relevant to our evaluation issues. The 
bad news is that those records may be deeply buried, unevenly 
maintained, or constructed in such a way that they yield unwieldy 
data. So, while you first consider available data to answer your 
questions, do so with care and with consideration of their accuracy, 
accessibility, completeness, and utility. 

Another strategy for gathering evidence i.i to consider: "Who 
might provide a credible answer to our questions?" Should we ask 
parents or teachers to report whether they have noticed any improve- 
ments in student attitude toward school? Should we interview some 
key students about how they feel about school? Do we ask counselors 
to provide anecdotes related to their meetings with students and 
about student attitudes toward school? Would community members 
be a good source of information for some of our questions? Does the 
district office have any records that might be useful? In other words, 
who are knowledgeable sources of information? 

Closely related to the issue of who there is to ask is that of how 
to get relevant information from them. There are many possible 
alternatives: 

• Should we administer questionnaires — to teachers, parents, 
students, community members, others? Should we use struc- 
tured or open-ended questions? 

• Should we conduct interviews? Should this be done individually 
or in small groups (often called "focus groups")? Will we use 
structured or open-ended questions? 

• What about attitude measures or other inventories? Do we try 
to make our own? Are there some we might adapt from other 
schools? Are there any that are commercially available that 
might be appropriate? 

• Would observations using checklists or open-ended techniques 
help us answer our questions? 

• What about students' work samples, portfolios, special perfor- 
mances, and so on? 

• Are there alternative measures that have been developed by 
other schools or districts that might suit our needs or could be 
adapted to them? 

• Are there other creative strategies for getting the information 
we need? What about videos or audiotaped interviews? 
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How can we get the 
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Select Tracking 
Strategies by 
Balancing 
Competing 
Requirements 

Credibility. In deciding which strategies are best, again, there is no single 
right answer. Issues of data quality and of credibility of information 
are certainly paramount, but, unfortunately, what is credible to one 
important constituency may not be so to another. For example, the 
general public and many parents put a lot of faith in standardized 
test scores whereas teachers who work with tests on a regular basis 
know the limitations of such test information. On the other hand, 
teacher observations of student work over a period of time can provide 
rich and credible information about student progress and status to 
other teachers and to school administrators but are difficult to sum- 
marize for outside audiences and sometimes viewed as suspect by 
them. Although, ideally, you want to be responsive to the needs of all 
of your important constituencies, the reality of available time, exper- 
tise, and measures will impose important limitations on your instru- 
ment selections. 

Validity. In addition to which measures can provide information that is 
both credible and feasible to collect, your selection of instruments 
must consider very carefully the concept of "validity." Validity is 
largely a question of whether a measure actually measures what it is 
intended to measure and whether its results can provide accurate 
inferences for the decision(s) at hand. In the case of our evaluation 
planning, a key validity concern is the match between the information 
provided by the instrument and what we want to know— the infer- 
ences we want to draw from the results. For example, if we are using 
results of a published, standardized mathematics test to assess our 
school's effectiveness in teaching mathematics, one important valid- 
ity issue is whether the content of the test reflects essential knowl- 
edge and understanding in mathematics; another is the match be- 
tween the test content a id our school curriculum. Without this match, 
the test provides information that essentially is irrelevant to our 
concerns. Similarly, in trying to track the process of implementing a 
new program at our school, validity demands that there be a good 
match between the intended key processes of your program and the 
specific items used to assess them. 

The sensitivity of your instruments to what you are trying to 
accomplish, in short, is critical to your assessment. It also is critical 
to your ability to demonstrate your success and to uncover true 
strengths and weaknesses. 
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Feasibility and ease of data collection therefore need to be care- 
fully balanced by concerns for validity. For example, using readily 
available standardized test score data and attendance data gleaned 
from last year's rosters could save a significant amount of time that 
would otherwise have to be devoted to instrument development or data 
collection. The price, however, could be insensitivity to school goals 
and intentions for the first measure and inaccuracy for the other. 

Recent changes in curriculum increase the possibility of mis- 
matches between curriculum and existing tests. Often, the two are 
grounded in different philosophies of learning and different psycho- 
logical theories. Some whole language advocates, for example, take 
issue with the design of standardized tests built on the theory that 
reading is the acquisition of a sequence of discrete skills. They believe 
that these tests cannot measure valued literacy skills. Many mathe- 
matics educators have expressed similar concerns about the content 
of many existing standardized mathematics tests. 

Innovative When the test and the program are not congruent, other indica- 
measures. tors of progress are a necessity. Student writing performance, the 
number of books read, work samples, or entire portfolios of work, 
parent or teacher observations, or parent satisfaction with student 
progress are some of the many possibilities that could be substituted 
for or added to standardized test scores. 

Growing concerns with existing standardized tests, in fact, have 
given rise to an explosion of interest in developing new, performance- 
based measures of student outcomes. As mentioned, these include 
student work samples, science demonstrations, extended projects, 
speeches, collections of essays and other work samples affectionately 
known as portfolios, and the like. Data from these sources have the 
advantage of providing actual samples of meaningful student work 
that can provide insights into student thinking processes. But work 
samples and work collections take time to judge. Criteria need to be 
developed and agreed upon, and scorers must be trained to apply 
scoring guidelines consistently across papers. Despite the time and 
effort, however, teachers often say that the time spent in defining 
program goals, in specifying performance expectations, in generating 
scoring criteria, and in actually scoring the samples is great staff 
development because it helps them to agree on what their programs 
are trying to do, and they can apply the methods to their own 
classrooms, A second by-product of collecting work samples is the 
identification of model student work that will become exemplars for 
teaching next year's students. 

Look for growing resources in the area of alternative measures 
of student performance. Many states, districts, and even schools, as 
well as research and development agencies and test publishers, cur- 
rently are involved in systematic development efforts. These will yield 
measures that can be adopted in full or adapted to local school needs. 
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Self-report Questionnaires and interviews are standard, tried-and-true meth- 
instruments. ods for directly asking about issues of interest. Questionnaires can be a 
relatively fast and cost-effective way to find out directly what you want 
to know, albeit sometimes at a somewhat superficial level. Often you 
can adapt a questionnaire used by another school for your own purposes 
at little cost. (Resources A and B at the end of the book include sample 
teacher and parent questionnaires from a middle school that you could 
adapt to suit your information needs.) Although it does take time to 
design clear questions and summarize results, as well as patient follow- 
up to assure adequate return rates, the result can be information that 
is tailored to your specific evaluation questions. 

Interviews, by phone or in person, have the common-sense design 
appeal of questionnaires and, at higher cost, can yield deeper under- 
standing of issues of interest. With trained and persistent interview- 
ers, you can assure adequate return rates and can bypass potential 
problems in respondents' reading ability. 

Observations. Formal observations are a little more intimidating to design 
because they require you to get very specific about the types of actions 
or types of behavior you are interested in or expect to see as a result 
of your program. They also require careful analysis of when, where, 
and how often you expect to see such actions; consideration of reason- 
able times for observation; and repeated observations to assure reli- 
ability of measurement. Conducted by trained and impartial observ- 
ers, however, systematic observations can yield highly credible data 
on what's actually happening in your school and its programs — rather 
than what people say is happening (the latter is colored by social 
desirability and by respondents' notions of what they think should be 
happening). 

Similar to interviews, observation require significant amounts 
of observer time and thorough training. Combined with the need for 
careful development and repeated observations to assure reliable re- 
sults, formal observations are a costly data collection strategy. Whether 
or not formal observations are part of our data collection strategy, our 
informal observations, gathered continuously over the program and 
focused specifically on programmatic areas of interest, are invaluable 
for making sense of results and refining program effectiveness. 

Unfortunately, concerns for feasibility and credibility often push 
us in opposing directions. Feasible often means inexpensive, "quick 
and dirty," something that can be accomplished by local staff on top of 
existing responsibilities— such as teacher-made or school-developed 
surveys that have not been subjected to technical scrutiny. If you are 
worried about the credibility of your results, you may have to bring 
in an independent evaluator to help with instrument development 
and validation processes. 



Feasibility and 
credibility 
trade-offs. 
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Use Multiple There is so such thing as a perfect instrument, and even the best 
Information ones reflect error as well as truth. Because of inherent errors in any 
Sources to instrument, it is highly advisable to use more than one source to 
Address Each g atner information related to each of your evaluation questions. The 
Question advantage lies both in the validity of your findings and in their 
credibility to various audiences. As summarized in Table 2.2, each 
information source has its weaknesses and is vulnerable to criticism 
by those not agreeing with its findings. If several sources provide 
similar results, confidence in your findings will be strengthened. 



What Standards At the same time you select ways to gather information related 
Will We Use? to your questions, you should be asking yourself and/or your advisory 
group, "What standards will indicate that we are making progress?" 
Although it may seem premature to be worrying now about standards 
for making judgments, it is not. Your standards will have important 
implications for what information you will collect and from whom, 
and therefore they are essential to devising an appropriate data 
collection strategy. 

Evaluators typically use one or more of the following strategies 
to find out whether programs are improving or need revision. 

Look for change Here, the primary issue often is whether things are improving; 
over time, if they are, we can consider ourselves successful. Furthermore, when 
we observe a trend over time, we can be more confident that some- 
thing is indeed occurring in our program than if we just take a 
one-time-only snapshot. Are student test scores, after adjusting for 
changes in student population or test norms, getting better? Are 
scores better than before we implemented our improvement plan? Are 
parents more satisfied with and involved in our school? Are new 
instructional techniques being used two years after the in-service? 

If change over time is important, we need to know where we stood 
prior to the start of a new program and then use the same or 
parallel measures at regular intervals to measure our progress. 



Look for Here, the issue is how our program and its outcomes compare 
improvement with tLose of others who are similar to us — or others who represent 
relative to some goals to which we aspire. Success can be defined in terms of a favorable 
comparison group, comparison. Are our students achieving at the same or better levels 

than students in similar schools? How do we compare with state or 
national norms? How do we compare with "distinguished" schools or 
other model programs? If a comparison is implied by our evaluation 
question, we must select an appropriate comparison group. 



TABLE 2.2 Selecting Evaluation Instruments 



Advantages 

Questionnaires Can probe several aspects of the program on one 

measure 

Can get candid, anonymous comments and 
suggestions if space provided for comments 

Questions are standardized for all respondents 

Questions can be made selected response for 
quick, machine scoring 

Gives respondents time to think 



Interviews Can do by phone at times convenient to respondents 

Allows people who can't read or write to answer 
Can be conducted in respondents' native 
language 

Flexible; can pursue unanticipated lines of query 
Depth; can probe responses 
Persistence can yield high return rates 

Observations Can use required observations (such as teacher 

evaluation) for other purposes 
Observers can see what teachers or others 
actually do, not what they say they do 



Limitations 



Not as flexible as interviews 

People often express themselves better 

orally than in writing 
Responding is often tedious and people 

forget to return questionnaires 
People may give "socially desirable" 

responses 
Requires literacy 

Depth of information sometimes sacrificed 
for breadth 

Time-consuming, thus costly 
Possible for interviewer to (consciously or 
unconsciously) influence responses 
People may give "socially desirable" 
responses 



Time needed to develop observation 

measure and train observers 
Presence of observers may influence 

classroom behavior 
Time needed to conduct a sufficient number 

of observations 
Scheduling problems 
Costly 

continued 



to 
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Advantages 



Limitations 



Performance Tests 
Essays 

Demonstrations 
Projects 
Performances 
(music, dance, 
drama, speech) 



Portfolios 



Archival Information 
Test scores from 

prior years 
Attendance 
Discipline records 
Teacher records 

Published Achieve- 
ment and Attitude 
Measures 



Provide actual sample of student work 
Can provide "diagnostic" information about 

student performance and about instruction 
Available for all subjects (unlike standardized tests) 
Credible method for assessing complex skills and 

processes 

Contextualized and relevant to real-life situations 



Provide a broader range of student work than 

one sample 
Can be used for many purposes 
Provide students opportunities to observe own 

growth and reflect on own work 
Encourage integration of instruction and 

assessment 

Records already exist 

No administration or development costs 



No development cost 

Information available about validity and reliability 
Most are quickly scored 
Clear methods for interpreting results 
Inexpensive to score 



Criteria for judging needs to reflect subject 
matter standards yet be understandable 
and usable by all 

Need many samples to draw conclusions 
about one individual; classroom/school 
inferences require fewer samples 

Scoring process is time-consuming 

Finding scorers may be difficult given the 
time commitment to training and scoring 

Costly if you pay scorers or use release time 

What constitutes a portfolio must be defined 
Scoring methods must be developed 
Costly to score 

Methods for interpreting need to be 
developed 



Gathering information takes clerical time 
May be incomplete, inconsistent 
May not have exactly the information you 
need 



May not match local goals 
Item formats often don't match real-life 
tasks 

Provide little diagnostic information 
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Against whom should we judge ourselves? Table 2.3 summarizes 
some possible comparisons that help answer typical evaluation questions. 

Try to achieve an Sometimes we have a very good idea of what specific goals we are 
absolute level of trying to accomplish, such as all students taking and passing certain 
performance courses or all students being able to communicate effectively in writing. 
on goals. Perhaps our goals are to have at least five teachers using whole 
language techniques in the primary grades or to have at least half of 
our students enrolled in Algebra 1 by grade 10. 

To assess whether we are meeting absolute standards, we need to 
consider what are reasonable yet "high* or positive expectations. Stan- 
dards need to be high enough to be meaningful but not unattainable. 

In reality, we most often use several approaches to standards in 
judging and reporting the success of our school. For example, we 
might have an absolute standard that we hope to achieve over time 
and will not be fully satisfied until we achieve it (e.g., all students 
successfully completing high school). Nonetheless, we will count our- 
selves successful if we make continuous and significant progress 
toward that standard. 

Worksheet 2.1 provides a format for recording the results of the 
session(s) in which you decide, by whatever means, what information 
you want to answer your evaluation questions, from whom and how you 
might get it, and your standards for success. With these tracking 
strategies firmly in hand, you are ready to schedule your data collection 
process, the central task in Step Three. (An example of tracking strate- 
gies for a whole-language program appears in Sample Worksheet 2.1.) 



Suggested Given our training and occupations, we had to exercise severe 
Readings restraint in our recommendati as for this section. With norm-referenced 
standardized tests under attack, alternative assessment guides are 
being produced exponentially. We urge you to attend meetings of your 
professional organization to keep on top of the quickly changing 
scene. 



Fowler, F. J. (1984). Survey research methods. Beverly Hills, CA: 
Sage. 

None of us escapes survey development. If you vant to know how 
to do it "right," use this guide. All bases are covered from sam- 
pling, to question development, to data analysis and ethical 
issues. 

Henerson, M. E., Morris, L. L., & Fitz-Gibbon, C. T. (1987). How to 
measure attitudes. Newbury Park, CA: Sage. 
Another essential from the Sage Program Evaluation Kit, this 
book deals with how to find or develop attitude measures as well 
as limitations in interpreting attitude data. 
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Sample Worksheet 2.1 

Tracking Strategies 


Evaluation 
Questions 


1. How far along are we in implementing integrated instruc- 
tion for social studies, language arts, math, and the arts? 

2. What support do teachers need to integrate instruction? 

3. Is student performance in reading, math, and writing 
improving in integrated instruction settings? 

4. Are special populations, special education, LEP, and 
gifted students being served appropriately? 

5. What has been the impact of our parent involvement 
program? 

6. What do we need to do to improve the program next year? 


Questions 


Evidence 
of Progress 


Source 
of Information 
(Instruments) 


Standards 
of Success 


1, 2,6 


Teachers using 
cooperative 
learning groups, 
student choices 
within activities, 
multiple groups, 
teachers coaching, 
subjects 

indistinguishable 


Informal 
observations 


All teachers using 
cooperative learn- 
ing; at least one 
"choice* activity 
per unit 


3,4 


Fail rates for 
special education 
and LEP students 


Competency test 


Fail rate in writing 
drops 


3,4 


Average percentile 
ranks in math, 
reading, written 
expression 


State assessment, 
Nationally 
standardized test 


Higher than last year 


5 


Number of parent 
volunteers 


Teachers' 
volunteer logs 


At least 10 hours 
a month 


5 


Attendance at 
parent education 
meetings 


Sign-in sheets 
for November, 
February, and 
June meetings 


At least 60 parents 
per meeting 


5 


Parent satisfaction 
rating of program 


Parent survey 

1 
i 


Satisfaction rating 
(average of survey 
questions 3, 4, 7) 
of 80% or higher 



Lindheim, E., Morris, L. L., & Fitz-Gibbon, C. T. (1987). How to 
measure performance and use tests. Newbury Park, CA; Sage 
A companion to the measuring attitudes book, this book serves 
as a valuable resource for selecting or developing measures. This 
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TABLE 2.3 Methods for Determining Standards of Success 



Evaluation Question 



Ways of Looking at Information to 
Determine Standards of Success 



Needs Assessment 

How are students doing? 
What are our strengths? 
What areas need 
improvement? 



Formative Evaluation 

Is the program being imple- 
mented as planned? 

How can we improve the 
program? 



Average student performance compared with 
average performance of students in schools 
like ours 

Proportion of students scoring in different 
quartile ranges or receiving each score on 
a performance test scoring rubric compared 
with the distribution of scores at similar 
schools 

Average scores compared with our perfor- 
mance in the past 

The distribution of scores/positive question- 
naire ratings compared with the past (3-5 
years) 

Average performance (mean or median score) 
compared with the state or national aver- 
age on the same measure(s) 

Average performance (mean or median) com- 
pared with a standard/goal set by the state, 
nation, or other agency (e.g., a "3" on ad- 
vanced placement exams is "passing*) 

Distributions of scores/proportions compared 
with state, national, or other standards 
(e,g M national goal that all adults will be 
literate) 

Percentage of content match of courses with 
required state frameworks or national cur- 
riculum association guidelines (e.g., NCTM 
Curriculum and Evaluation Standards) 

Match of curriculum as implemented to 
intended implementation plan as described 
by a funding proposal, monitoring agency, 
or school program description 

Comparison of classroom observations with 
what staff says they are doing 

Comparison of program activities/ 
outcomes with community expectations as 
described in parent meetings, kinds of par- 
ent concerns about the program (e.g., com- 
plaints about not enough homework) 

Comparison of program with similar programs 
that have been identified as "exemplars" 
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volume of the Program Evaluation Kit provides sources for track- 
ing down published tests as well as steps in developing perfor- 
mance tests. Of interest is the section listing sources for pub- 
lished tests in a wide range of fields from vocational education, 
to foreign language, home economics, and physical education. 

Popham, W. J. (1981). Modern educational measurement. Englewood 
Cliffs, NJ: Prentice-Hall. 

Written in the same lively, engaging style as his evaluation book, 
Popham's measurement text provides the conceptual foundation 
and an overview of test development procedures for the major kinds 
of assessment devices: criterion-referenced tests, norm-referenced 
tests, attitude surveys, and essays. Although not a step-by-step 
manual, the book explains central concepts in measurement 
clearly and helps evaluators understand the strengths and lim- 
itations of their tools. 

Vierney, R. J., Carter, M. A., & Desai, L. E. (1991). Portfolio assess- 
ment in the reading-writing classroom. Norwood, MA: Christo- 
pher-Gordon. 

This is one of the first of the surely many to come dealing with 
portfolio assessment. Although the examples are from English 
language arts, they apply throughout the curriculum. Of particular 
interest to the evaluator are chapters on student self-assessment 
and portfolio analysis and record keeping. 



Step Three 

Manage Instrument Development 
and Data Collection 



In Step Three, you will select or develop instruments for gath- 
ering the data you need to answer your evaluation questions 
and then proceed to collect information from your population 
or sample of interest The central tasks in this step are plan- 
ning, scheduling, and monitoring. Thorough planning is a 
necessity as Murphy, of Murphy's Law fame, most certainly is 
the patron saint of research and evaluation! 



With your tracking strategies in hand from Worksheet 2.1, you are 
ready to select or create the measures you need and then use them to 
collect your evaluation information. The most efficient strategy for 
managing the numerous tasks involved with instrument acquisition 
and data collection is to create a management plan indicating the 
tasks that need to be accomplished, by whom, and when. Quality data 
collection requires monitoring. How you record your management 
plan, whether it is penciled in on your school calendar or whether it 
is a stand-alone document, is not as important as thinking ahead 
about how you will accomplish the tasks of selecting or developing 
instruments and of collecting information to answer your evaluation 
questions. This step is labor intensive, is time-consuming, and needs 
to be supervised by someone with an eye for details and a strong 
streak of perfectionism. 
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Consolidate In Step Two, you were encouraged to brainstorm ways to gather 
Your Evaluation information and advised to use at least two different methods, when- 
Interests ever possible, to strengthen the credibility of your results. The work- 
sheet you used resulted in an evaluation plan that links specific 
tracking strategies to each of your evaluation questions. As you get 
down to the nuts and bolts of data collection, you will need to reor- 
ganize your plan and shift your focus from the question: "What in- 
struments do I need to address each of my evaluation questions?" to 
"How should the information I need be distributed by instrument?" 
Now is the time to review your planned tracking strategies and 
consolidate them into a feasible number of specific data collection 
instruments. Look across the information sources you have desig- 
nated for all of your evaluation questions and find those that fre- 
quently recur. You may find, for example, that, while you have several 
pages of information sources when listed for each question separately, 
they really boil down to a teacher questionnaire and a parent ques- 
tionnaire — each addressing a variety of specific issues — and an anal- 
ysis of various student records. If you have followed our advice on 
triangulating information sources, you probably also will have iden- 
tified some issues that are common to several instruments (e.g., 
planning to ask both parents and teachers about their satisfaction 
with students' progress in reading, math, liking of school, self-concept, 
and so on). 

Sometimes you will find that you have designated a tracking 
strategy that is only relevant to one evaluation question or to one or 
two specific concerns within a single area. If this is the case, then you 
may want to reassess whether this unique data collection strategy is 
really necessary or whether needed information could be collected 
nearly as well through another strategy you're planning to use or 
through a more cost-effective strategy. For example, suppose you 
have planned to do a parent survey to address numerous evaluation 
questions and have designated a community survey to answer a single 
question about available opportunities for community service and 
about satisfaction with student preparation. If these are critically 
important issues to your evaluation or if you feel that surveying the 
community will send an important message to them, then, by all 
means, stick to your plans. If, however, these are relatively peripheral 
issues, then you may want to address these issues through your 
parent survey and a few well-placed calls to community service 
agencies. (You might also wrack your brain about whether such 
information may be available through other means — for example, 
community agencies may already have gathered data on service 
opportunities.) 
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Create a 
Blueprint for 
Instrument 
Development or 
Instrument 
Selection 



Worksheet 3.1 asks you to designate, across all your evaluation 
questions, all the issues that need to be addressed by each data collection 
strategy. For example, in reviewing Worksheet 2.1, you might find that 
you designated a parent questionnaire for gathering information rele- 
vant to questions 1, 2, and 3 and listed specific indications for each; 
here, in Worksheet 3.1, you'll want to transfer into one place all the 
indicators that are planned for the parent questionnaire. In reorganiz- 
ing your evaluation questions by the data collection instruments needed 
to answer them, you will have created a basic blueprint for each instru- 
ment. This instrument blueprint will be an invaluable tool for selecting 
or developing appropriate instruments. 

Remember that the match between your instruments and the 
issues you plan to address is a critical requirement for validity. If your 
plan is to select from among available measures, the blueprint will 
help you to find the best fit. If, on the other hand, you plan to develop 
your own customized instruments, the blueprint will help you stay 
focused and write or adapt items that meet your primary and specific 
interests. Whether you select or develop instruments, you will need 
to pay attention to a range of factors that influence data quality, such 
as reliability, test administration considerations, appropriateness for 
your population, and content quality. 



Create a A consolidated list of needed instruments is also a good starting 
Consolidated place for your management plan. By designating when you plan to 
Management Plan use ea °h instrument, the steps you will need to complete to be prepared 

for such use, and who is to be responsible for each step, you help to 
assure a smooth (or at least as smooth as possible) development and 
data collection process. Your schedule will be backward chained from 
when mandated reports are due or from when it makes sense to collect 
particular kinds of information. If you're trying to assess end of the 
year progress, it doesn't make sense to collect the information in 
February! 

The steps in your plan will vary somewhat depending on whether 
you plan to use existing data, to collect information using available 
instruments, or to create your own instruments. Bear in mind the 
following: 

• If you need to create your own measures, you will need a time 
line for development, review, pilot testing, and revision. (See 
Table 3.1 for a sample process.) Be sure to leave adequate time 
for repeated proofing and for quantity printing. 



Sample Worksheet 3.1 

Instrument Blueprints 


Instrument 




Portfolios 


What kinds of mathematics problem representations did the students use? 

What percentage of the assignments represented critical thinking/problem solving? 

What difficulties with math did students identify in their self-evaluations? 

\Vnflt IcinnR ftf Wftrlc did atiidnnfa inrlnHo frvr ftmiv* onlf aolonf a/1 oamnlno? 

ft 11 civ xwiiiuo ui wuiiw uiu otuuciito tiiLiuuc ior tneir sen-seiecteu samples' 
What were the range of strategies students used to solve problems? 


Parent Questionnaire 


Do parents notice a difference in their child's attitude toward mathematics? 
What kinds of help do parents provide? 

\W nflf. KiTinfl nf flnnif iahaI Holn Ar\ naronfa aotr ofurlanfn ...J 1 ? 
TTiiai/ xwiuuo ui auuiLiuiial llciy UU paicllts Say StUQeutS need; 

Do parents understand the purpose of the new mathematics program? 


Student Questionnaire 


Do students like math? 

Do students feel competent in math? 

Where do students say they would like more help? 

Arc ton stndpnts Hpinc pVmllpncrpd? 


Faculty Interviews 


What difficulties are faculty having setting up "han Is-on" lessons? 

In what ways are faculty integrating mathematics instruction with other subjects? 

What additional support would faculty like? 

How do faculty rate the mathematics resources (pattern blocks, beans, rice, and so on)? 

Do facultv rintirp anv diffprpnrPQ in ofudonf affifnrloo? AKilif*r f ^ u oon n«%9 

*-* la^iAAi/jr iiui/ivc any uiiiciciitcg ill oLUUcllL UVI/ltuQcSr rVDlllty tO Keep Up. 


Standardized 
Nationally Normed 


What are student average standard scores in math problem solving? Concepts? 
What are student average standard scores in these areas for special education, gifted, 
and T^KP atndpnts? 

What items pose the most difficulty? 

What kinds of mistakes do student? tend to make on selected response tests ana why? 


State Assessment 


What Kinds Hi PinSWPT*S did nil v atiidonto cri fViA nnon an An A «v%rk4>Vt r*m%*\all amO 

" uui ftuiu ° *** auoTYcio uiu uui ntuutfiito give v> tiic open-enueu matn Question; 

What is our average score in math problem solving? 

How did kids do in geometry concepts? In number concepts? 


Competency 


What strategies did students use when answering the competency essay about solving the 

open-ended problem? 
What was the average percentage getting the five problems correct? 


Other 


What kinds of questions appear on teacher-made, end of unit tests? 

What kinds of teacher-developed questions gave students the most difficulty? 

How does progress on classroom tests compare with portfolio ratings and standardized tests? 
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• Where you are using commercial instruments or those developed 
by others, be sure to leave yourself adequate time for ordering, 
shipping, and distributing materials. 

• If you are collecting existing information (e.g., information 
from student cumulative records), you will need to create user- 
friendly and summary-friendly forms for recording the data. 

• If data collection requires special skills or orientation, be sure 
to leave adequate time to develop training for and deliver it to 
data collectors. 

Lest the instrument development or selection process sound too 
complicated, remember that much of your information probably will 
come from already existing measures, such as mandated tests, teacher- 
assigned student work samples, or projects; from routinely collected 
data such as attendance, textbook expenditure, or counseling refer- 
rals; or from adapted surveys that spring rather easily from your 
specific evaluation questions. For most of us, "instrument develop- 
ment" means finding a parent questionnaire or interview used by 
another school or district and adapting it for our own purposes. The 
time line for adaptation, also provided in Table 3.1, is considerably 
shorter than that for development "from scratch. n And, for those of 
you wishing to get a head start, we have included sample parent, 
teacher, and student questionnaires in Resources A, B, and C. Adapt 
these if they fit your situation and, as you encounter examples from 
other schools, add them to these questionnaires for later reference. 

Having determined the general tasks that need to be completed, 
you need to decide who will have primary responsibility for each as 
well as the important milestones for scheduling. This task-person 
time line becomes your data collection management plan. But, before 
you formalize this plan, you will need to consider who will be involved 
in administering measures (and whether they will need to be hired 
and/or trained); whether you will administer your instruments to a 
sample or to the entire population; whether your plan requires a 
comparison group for whom data must be collected; and when the 
optimal time to collect the data is. 

Sample Worksheet 3.2 presents a sample management plan. It 
contains the key information that the evaluation coordinator needs 
at a glance: Who is getting what from whom and when? The plan also 
helps you to identify potential bottlenecks in the data collection 
process. October, November, and April appear to be busy months. 
Should you reschedule some of your April evaluation activities — es- 
pecially in light of some of the other activities that you know will be 
occurring at your school? Should you begin instrument development 
a little earlier or later? Should you consider purchasing or adapting 
extant measures to hasten the development process (and lighten the 
burden)? Your data collection plan will not answer these questions, 

te\t continued on page 54 



TABLE 3,1 Data Collection Time Lines for Commonly Used Evaluation Instruments 



Time 



6 months prior 



3 months prior 



1 month prior 



2 weeks prior 



1 week prior 



Data Received 



Already Published 
Tests 



Type of Instrument 
Archival Information 
(attendance; past scores; School Developed: 
CUM information) Selected Response 



School-Developed Performance 
(includes observations, 
projects, essays, portfolios) 



Take inventory Locate records; 

Order needed materials determine what's 

missing and how to 
"fill in" if possible 



Preslug answer sheets 
with student ID, grade, 
other needed info 



Discuss uniform test 
preparation guidelines 
with teachers; explain 
content, consequences; 
and discuss ethical 
preparation guidelines 

Prepare test 

administration 

directions 

Distribute testing 
materials to teachers; 
go over administration 
procedures; explain 
importance of accurate 
information 

Organize answer sheets 
for scoring; check 
accuracy of special 
codes for processing 
reportr 



Monitor filing of 
current data 



Meet with advisory 
group to determine 
exactly what information 
is needed (grade, 
sex, teacher, month, 
and so on); assign 
codes for these items 

Practice summarizing 
data to iron out bugs 
in coding procedures 

Hand out data entry 
coding sheets to 
volunteers or other 
data collectors and 
explain procedures 

Make photocopy of data 
for "backup* and get 
data entered into 
computer or summarized 



If starting from scratch, 
develop questions and 
response alternatives; 
try out with sample, 
revise 

Adapt already existing 
survey; try out survey 
items with sample; 
revise; translate and 
try with bilingual 
sample 

Write cover letter for 
mailed surveys; write 
administration 
directions for tests, 
school surveys 



Run mailing labels; 
print surveys/tests; 
prepare class sets 
for teachers 

Mail surveys; 
meet with teachers 
administering 
tests/surveys 



Follow up and track 
down missing data 



Preferably a year prior, 
convene a committee 
to discuss tasks and 
scoring criteria; pilot tasks, 
scoring, and rater training 

Develop administration 
procedures; develop 
descriptions for students to 
understand task 
(sample items and 
scoring criteria) 

Produce materials; 
count and distribute; 
meet with teachers 
to explain administration 
procedures 



Check to see that 
answer sheets, 
portfolios, observation 
forms have codes 
for grade, name, and so on 



ox 

CO 



Task by Instrument 



Instrument: Math Portfolios 

Select group to develop/choose 

Develop tasks and criteria 

Produce 

Distribute 

Administer 

Score 

Analyze and interpret 



Instrument: Math Chapter Tests 
Select group to develop/choose 
Develop tasks and criteria 
Produce 
Distribute 
Administer 
Score 

Analyze and interpret 



Sample Worksheet 3.2 

Data Collection Management Plan 



Person Responsible 



Principal, mentor 
teacher, district 
evaluation office 



Head teacher: 
grades 3, 4, 5 



Instrument: State Assessment 

Select group to develop/choose 

Develop tasks and criteri 

Produce 

Distribute 

Administer 

Score 

Analyze and interpret 



Instrument: ITBS 

Select group to develop/choose 

Develop tasks and criteria 

Produce 

Distribute 

Administer 

Score 

Analyze and interpret 



District testing 
office 



District testing 
office 



Sampling 



Random — 10% 
grades 3, 4, 5 



Copies of ail 
tests; all grades 



Sept. 



Oct 



Nov. 



Dec. 



Jan. 



Feb. 



Mar. 



Apr. 



May 



All grade 4 



All grades 
K-5 
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but it will keep them from sneaking up on you and sabotaging your 
entire operation. Our sample data collection plan was developed after 
consideration of the following issues: 

Delegate Very often, it is a simple matter to decide who should be involved 
responsibility for in collecting information. When you want responses or work samples 
data collection, from students, teachers are the natural data collectors — unless there's 
some reason to believe that their presence will distort student re- 
sponses. For example, it is not a good idea to have teachers administer 
an attitude measure to students that focuses on students' attitude 
toward their teachers! Data collected from teachers most probably will 
be coordinated by an administrator, department head, or union repre- 
sentative, depending on the usual procedures in your district or school. 
The student council could be responsible for distributing and collecting 
nonconfidential student questionnaires. PTA volunteers, parr profes- 
sionals, or other adults affiliated with the school may be asked to 
conduct telephone interviews with parents. Classroom observations fall 
into the administrator's domain, but mentor tesjhers, union-approved 
representatives, district office staff, or local graduate students may be 
recruited if these are acceptable procedures in your particular setting. 

When selecting data collection coordinators, you need to select 
people who understand the need for objective, representative data, 
who can be expected to be impartial, and who will not try to influence 
respondents (including students) in any particular way. Your assis- 
tants should be nonthreatening to respondents. The truant officer is 
probably not a good choice for interviewing students with attendance 
problems. It also helps when data collectors are also well organized and 
able to handle the distribution and collection of materials without 
supervision and willing to follow up repeatedly to get the information 
or responses that are needed. Those who are doing parent telephone 
interviews, for example, should be prepared to call a number of times 
to find someone at home and with time to respond. 

Decide whether Your management plan for data collection identified what infor- 
to sample, mation you will collect by whom. At this point, you need to decide 
whether you will collect information from or about everyone or whether 
you will sample— that is, randomly select a proportion of the popula- 
tion of interest to be the subjects of your data collection. Should you 
mail a questionnaire to all parents or o^ ly a sample of them? Should 
you observe all teachers once a month or a sample of teachers on a 
more frequent basis? Should you collect work samples or tests from 
all students? Do you want to distribute the testing burden so that all 
students take some assessments but no student has to take them all? 
Do you want to distribute the respondent burden so that half the 
parents get called for interviews but each is administered only one of 
two interview forms enabling you to find out about a broader range 
of issues without overly taxing the patience or time of individual 
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respondents? As a general rule, having more respondents or more 
data points at the school level provides a better technical base for 
inferences, but concerns about technical quality must be balanced by 
ever-present concerns about limited resources — the resources of teacher 
and student time as well as direct dollar expenditures for instrument 
purchase/duplication, scoring, and other costs. 

Although there are ways to technically compute the proportion 
of total respondents you should have for valid inferences, common 
sense about what is credible and reasonable goes a long way when 
making such decisions. If the potential number of respondents for a 
questionnaire or survey is fewer than 20, it makes little sense to sample. 
At the other extreme, if there are 500 families at your school, inter- 
viewing them all is probably out of the question (although sending a 
survey home to them all probably isn't). What is a reasonable sample? 
Balance feasibility and credibility. Ask yourself, "Would I believe the 
results if they were based on only [half, a third, a quarter] of the school 
population?" 

Also, be sure to think about the number of respondents you will 
be dealing with as you design any new instruments. For example, if 
you expect to get responses back from 250 parents, you will want to 
be sure that their responses are easy to score and summarize. For 
sample sizes this large, you probably will want to have a majority of 
your q?iestions in selected response format, where respondents indi- 
cate a response on some given scale — agree to disagree, very effective to 
very ineffective, very satisfied to very unsatisfied — or choose from 
among given respond alternatives. You also will want to investigate 
the cost of selected response questionnaires that can be used with a 
Scan-tron or optical scanner. You may purchase "general purpose" 
questionnaire forms for both machines that will allow your equipment 
to summarize questionnaire data (as distinct from test data). Even if 
you have large samples and are relying primarily on selected response 
items, be sure to include a few open-ended questions to give you a 
better sense of what people are feeling and why. 

Schedule data Consult your school and district master calendar as you schedule 
collection, data collection. Scheduling considerations include the following: 

♦ When are accountability reports due? 

• When will the information we need be available? 

♦ How much time do we need to design, try out, and administer 
measures? 

• How much time will it take to summarize, analyze, and report 
the data? 

For example, suppose you have to report to your school board at the end 
of the fiscal year in June about how you spent your school's "innovative 
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program" allocation as well as about the program's results. You plan 
to use teacher and student questionnaires that your faculty and par- 
ent advisory committee will create. You plan to give yourself at least 
a month to summarize and discuss the meaning of your results, and 
you need at least three weeks to send out and collect surveys and to 
follow up on late or missing respondents. You also want to collect the 
information as late as possible in the school year to give your new 
program the maximum time "to take," but you don't want to run into 
the Open House-May testing crunch. You decide, therefore, that late April 
is a good time for your surveys. Working backward from your tenta- 
tive administration date, you designate time for the development of 
draft questionnaires and time for review, revision, duplication, and 
development of directions for administration. You now find you must 
begin your first task in October. If you're planning to do complex in- 
terviews or to conduct formal observations, you also will need to build 
in time to develop or find the instrument, to recruit and to train your 
data collectors, and to give them opportunities to practice using the 
instrument. 

If you're relying on information summarized by others, of course, 
you're dependent on their schedules, which too often don't match yours. 
Ir. these cases, you just have to do the best you can, which sometimes 
means delaying a desired report, submitting a draft that is missing a 
section, or using available data from prior years. 

Cost: An As you do your scheduling and planning, you want to keep in 
ever-present mind the costs of enacting them. Direct costs are fairly easy to 
mediator, estimate. Publishers can provide you with per student costs for test 
administration and various scoring options. Printers can give you per 
page copying costs for bulk printing. Costs of mailing out and provid- 
ing return envelopes for parent surveys can be estimated based on 
current postal rates. Costs of t-avel/mileage getting to and from 
needed data collection sites car be estimated based on prevailing 
district or school policy. If you have to hire special staff to complete 
your plans, estimate how much time you will need from them and 
what their hourly or daily rates are. Special consultants easily can be 
asked for budgets detailing the costs of their involvement. Scoring 
and analysis may require the purchase of new software, as may the 
production of appealing reports. Every item in your time line probably 
brings with it direct costs. Be sure to be generous in your estimates — 
everything always seems to take much, much more time and money 
than we ever expect. 

But direct costs are only a part of the picture. Opportunity costs 
represent another important area of consideration. Using existing 
staff or even volunteers to work on the evaluation means they are 
using their time on this and not doing something else. Do they have 
sufficient available time to do all that is required? Student time also 
is a major concern, as student time spent in data collection often is 
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time diverted from instruction. This concern may be diminished immea- 
surably in the case of "alternative assessment" or 'performance assess- 
ment," where the assessment tasks themselves are thought to be mean- 
ingful learning opportunities, often indistinguishable from instruction. 

The point is, once you have your plan in hand, you need to 
consider the costs implied by the plan. Are they reasonable? Are they 
feasible? If not, you may find you need to rescope your planned effort: 
delay some issues until next year, swit'h to some lower-cost data 
collection strategies, collaborate with other schools or districts to 
reduce instrument development costs, or entice your local university 
t? give students course credit for helping with your effort. Be creative 
to cut costs and maximize efficiency! Table 3.2 provides some sample 
costs of common evaluation needs. 



Avoiding Data Data collection brings with it many challenges, and, as mentioned 
Collection in the introduction to this step, it often seems that Murphy's Law is the 
Disasters operant rule: Whatever can go wrong, will! Advance planning and atten- 
tion to frequent pitfalls, however, can greatly diminish your problems. 

Review measures Consistent measurement depends upon well-written, unar<I>igu- 
for clarity, ous questionnaire, interview, or test items. Try to anticipate your 
relevance, and bias, respondents and the many ways they could misunderstand a test item 

or question. Be particularly vigilant about attitude survey items that 
have different meanings in different cultures. For example, we ad- 
miristered a well-known school attitude measure that awarded points 
for "high self-esteem" for items in which students felt they should 
speak out in class and make their opinions known or in which stu- 
dents felt they did well or were outstanding in some way. A Chinese 
graduate student reviewing the instrument mentioned that the well- 
adjusted Chinese student would lose points on these self-esteem 
items because "putting yourself forward" was not considered appro- 
priate in China. 

Although you could literally second-guess every item on every 
measure you use, do try to get others with different educational levels, 
cultural perspectives, or reading ability to review your instruments 
so that you are presenting all your respondents with an identical task, 
a task they will interpret similarly and that will carry the same 
meaning. People within a school tend to use the same jargon, such as 
restructuring, literacy, developmental^ appropriate, or at risk. These 
words, however, carry different meanings for some audiences and no 
meaning at all for others. It is much better to discover a "bad question" 
before data collection than to lose information you think is important. 
One of us is involved with an innovative curriculum development- 
teacher training project and very interested in whether the project 
has affected the ways in which teachers judge students. On the 
teacher questionnaire, we asked, "I would use the following observable 
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TABiuE 3.2 Typical Costs Associated With Evaluation 



Service or Product 



Evaluation consultant 



Data collector: phone calls, 
observations, coding archival 
data 

Published test materials 
Published test scoring 



Outside scoring for local 
selected response test 

Scoring for performance 
tests using teachers 



Data analysis expert 



Cost 



County office: free 
District office: free 
University: $250/day 
Independent: $35-$150/hr. 

$10-$12.50/hr. for a university student 



Booklet of items: $3-$5 each 
Answer sheets: $0.12-$0.15 each 

Scoring service: $.92 for list of scores 
Additional reports: vary from $0.10 to $1.75 
per student 

Set up charge: $200 

Scoring and simple reports: $1.75/student 

Training: minimum 3 hours per essay type 
depending upon complexity of rubric. 
Rating: 1 minute per paper for holistic ru- 
bric and short essays; 2-3 for longer; up to 
5 minutes per paper for multi-scaled ru- 
brics. Calculate time needed by multiply- 
ing number of paper/tasks by time per task, 
then hours for training to get total time 
needed. Convert to hours. Divide by num- 
ber of teachers rating to get person-hours. 
Multiply by teacher hourly rate or cost of 
substitute per hour to get total cost. (Faint!) 

Same as rates for evaluation consultant; if 
data are on a computer disk, cost will be 
significantly reduced 



indicators to measure what my students have learned . . «" and found 
that literally all teachers — project and nonproject— checked e rr ery 
single one of the indicators listed. We really wanted to know whether 
project teachers value a wider variety of indicators for evaluating (i.e., 
grading) students. We should have asked: "I use the following observ- 
able indicators for grades,* followed up with a question as to how each 
was weighted. Until we saw the teacher responses, it did not occur to 
us that we were asking the wrong question. Had we tested the item 
with a few teachers, we would not have lost data by using this loosely 
worded question. 

Building in a review process for your measures not only mini- 
mizes different interpretations of the same question, it also helps you 
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find potentially biased or offensive questions. Biased questions are 
difficult to identify a priori (after all, who of us intends to be biased?) 
and often don't show up until after respondents have returned their 
answers. A biased item is one to which students with identical ability 
levels and similar exposure to instruction respond differently. These 
items generally are few in number. Offensive questions, on the other 
hand, can be spotted beforehand by knowledgeable reviewers. Items 
or questions that present racial or sexual stereotypes or that delve 
into potentially embarrassing issues should be revised. For example, 
in your review of the dropout program, you want to know what kind 
of support parents can provide for theii child's education. You ask 
them how many years of education they have had. If your parent 
community consists of recently arrived, economically disadvantaged 
immigrants, you will be asking a question that parents may find 
embarrassing to answer. 

Prepare clear You will also need to have clear directions to respondents and for 
directions, people administering student measures or conducting observations to 
standardize administration conditions. You want all people respond- 
ing to the same measure or being observed to understand the task in 
the same way so that differences in responses will reflect real differ- 
ences among people and not simply different interpretations of the 
same task. Words such as "indicate" are not as clear as instructions 
to "circle the response" or "write your answer in two sentences." 
Remember that, while you may understand quite well who is to be 
sampled and how, the respondents will find loopholes and ambiguities 
that you never anticipated. For example, the directions to teachers 
on the California Assessment Program say: "Limited English Profi- 
cient students who are not instructed with grade level materials are 
exempt from testing." We received numerous questions about that 
direction: What about LEP students who clearly can't read but who 
are instructed with grade-level materials in a "sheltered English" 
situation? What about students who are reading materials designated 
for the grade level but who are at the low end of the difficulty level? 
What about the LEP students who are placed in regular classes because 
there are no special programs available but who can't really under- 
stand what's happening? Whenever possible, have a teacher, prefer- 
ably your most contentious, read data collection directions and iden- 
tify ambiguities. 

Too many of us give a cursory glance at test/questionnaire direc- 
tions and think, "Oh, this is just like last year." Those who don't give 
a cursory glance wait until administration time to read directions. If 
teachers are administering some "standardized" task— an essay, a 
group discussion, or even a state-mandated test— it is wise to go 
through a simulated administration prior to data collection, having 
some participants role-playing unruly or inattentive students, to 



7 - 1 
ERIC 



Thoroughly orient 
and train data 
collectors. 



TRACKING YOUR SCHOOL'S SUCCESS 



point out administration procedures that are tricky, ambiguous, or in 
need of special attention. You'd be surprised, for example, at how many 
teachers cannot answer student questions about how much time is 
allotted for testing, or whether guessing is encouraged/allowed, or 
whether they should write in pen or pencil. If you are collecting 
information that will be machine scored and students must be pro- 
vided codes for their language status, identification numbers, or the 
like, it is crucial to get this information to teachers well in advance 
and to remind them to check student answer sheets for accuracy. 

If data collection involves observations, essays, student perfor- 
mances or products, or extended interviews, the data collectors need to 
know the conditions under which each is to be collected. Should all class 
observations occur during the morning when "skills" are taught? Should 
teachers be told in advance that the observers are coming? Are students 
allowed any assistance while producing essays, projects, or performances? 
Are dictionaries allowed? Calculators? Cue cards? What are the time 
limits? Should students have names or just identification numbers? Ask 
yourself what variables other than the program could explain perfor- 
mance differences among classrooms and try to eliminate as many as 
are realistic through standardized conditions. Use administrators who 
fully understand and will enforce those conditions. 

Many observations and open-ended interviews, furthermore, re- 
quire data collectors to be extensively trained — to make sure they 
understand their tasks as well as how to code and follow up on 
particular responses and how to reliably and consistently use given 
instruments. Training in these cases requires thorough orientation 
to the purposes of the instrument, the meaning of its questions or 
categories, illustrations, and practical examples of applying the in- 
strument, as well as plenty of practice, and actually a final test to 
verify that the data collectors know how to appropriately and cor- 
rectly use the instrument. 

Motivate If you want the information you gather to be useful, you must 
respondents when organize and think though your data collection effort well in advance 
necessary, of actually administering instruments. Because people at the school 
level are not always highly motivated to take time out of a busy 
schedule to respond to questionnaires or interviews, you will need to 
prepare a convincing statement of the importance of respondent 
participation in the data collection effort. We have found that explain- 
ing the purpose of a teacher questionnaire at a brief faculty meeting, 
then providing time for teachers to respond and return their ques- 
tionnaires on the spot, has resulted in a better return rate. 

To increase parent survey response, some principals reward 
classrooms that have 90% return by allowing students first place in 
the lunch line or a free ice cream much as has been done during PTA 
membership drives. Collecting data from small children (grades K-3) 
generally involves no more motivating than to say, "We would like . . ., 
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please do your best." When tasks might be difficult for young children, 
they need to be reassured that the task is indeed difficult and that 
they are doing well. 

Middle and high school students often take data collection more 
seriously when a researcher visits and asks for their cooperation (and 
candid opinions); it would appear that the presence of an authorita- 
tive outsider sometimes invokes "company" behavior. Depending upon 
the situation at your school, student council members could ask students 
to complete surveys. Achievement and attitude measures that could 
raise student anxiety or that are logically part of the instructional 
sequence are best administered by the teacher. 

Better yet are evaluative devices that are built into instruction 
such as journals, logs, writing pieces, or group discussions that ask 
students to reflect upon what they have learned, how they are learn- 
ing, what difficulties they are having in a particular area, and what 
their triumphs are. Monitoring these assignments periodically will 
provide important feedback for prop m improvement and for track- 
ing the development of independent learners. 



Monitor return 
rates and follow up. 



And, when other approaches fail, persistence pays off. Plan to set 
up tracking systems to monitor carefully what data are received and 
what are still outstanding, instrument by instrument. Then, follow 
up, follow up, follow up. If your first request for information does not 
yield a response, send out a second, even a third, and fourth request. 
Your persistence communicates the importance of the effort (and may 
enhance "guilt" responses). You want return rates as close to 100% as 
possible. Otherwise, you don't know whether your responses really 
reflect the population, or whether they reflect mostly the complainers, 
the pollyannas, or what. 



Anticipate 
potential snags. 



As "data day" approaches, review your management plan and be 
sure that materials and procedures are in order. If this is your first 
experience in planning a comprehensive evaluation, review the time 
lines associated with using various kinds of measures (Table 3.2) and 
say to yourself, "If anything can go wrong, it will." We have run into 
and created our share of data collection disasters. Perhaps a list of 
some of our more important oversights will help you cut your own 
losses. These disasters include 



• forgetting to ask students to put their names on questionnaires 
and tests, which made it impossible for us to match student 
attitudes with achievement; 

• ordering the wrong version of a primary grade battery, a version 
that did not include the subtest we needed to gather informa- 
tion about LEP student language use; 
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» not double-checking our test inventory and running short of 
answer sheets, causing us to Xerox extras and then later trans- 
fer student responses, by hand, to the machine-scorable sheets; 

• scheduling data collection for the day after Halloween and the 
day before Memorial Day; 

• supervising a teacher committee developing an algebra screen- 
ing exam under such highly politicized circumstances that 
items could not be revised even though data indicated they 
were poor; and 

• not reading test administration directions far enough in ad- 
vance to be able to answer teacher questions in time for admin- 
istering the test. 

When all else fails, On the other hand, we have avoided many problems by making 
call for help, quick calls to friends at other schools for advice, for a rundown of the 
instrument administration procedures, or for extra materials. We 
have even called our friends in the state's Department of Education 
to deal with problems arising during administration of state tests. 
And we have memorized the test publisher's 800 number for request- 
ing an overnight shipment of test materials (although this strategy 
works best when you're in the same time zone). 



Suggested As important as knowing the nuts and bolts of instrument devel- 
Readings opment and data collection is a grasp of the reasons why certain proce- 
dures are important. Thus a knowledge of how data collection designs 
reduce competing explanations for your findings and strategies for 
making evaluation plans are useful. 

Fitz-Gibbon, C. T., & Morris, L. L. (1987). How to design a program 
evaluation. Newbury Park, CA: Sage. 

If you like the thought of countering alternative explanations for 
any evaluation findings you unearth, this is the book for you. It 
describes sampling methods and research designs (data collec- 
tion methods) that allow you to determine just which effects you 
can attribute to your program as opposed to one-time favorable 
circumstances (such as "the students were smart."). 

Herman, J. L., Morris, L. L., & Fitz-Gibbon, C. T. (1987). Evaluator's 
handbook. Newbury Park, CA: Sage. 

A book providing the "big picture" for those in a hurry. Flow- 
charts, summaries, and bulleted information lead the adminis- 
trator through evaluation models and the steps involved in each 
aspect of evaluation. The model evaluation report and the check- 
list format make this indispensable for the administrator hiring 
an evaluation consultant. 
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King, J. A., Morris, L. L., & Fitz-Gibbon, C. T. (1987). How to assess 
program implementation. Newbury Park, CA: Sage. 
This kit hits the mark for school-based evaluation with three chap- 
ters on data collection, including such topics as how to plan, how to 
get information from existing records, questionnaires, interviews, 

; and observation. It is clear and applicable in the school setting. 

Stecher, B. M., & Davis, W. A. (1987). How to focus an evaluation. 
Newbury Park, CA: Sage. 

Already mentioned earlier, the chapter on how to formulate an 
evaluation plan comes in useful now. It is especially helpful when 
you've hired a consultant and want to know what to ask. 
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What You 
Should Know 
Before Scoring 

Multiple choice 
and selected 
response scores. 



Step Four 

Scoring and Summarizing Data 



Your task in Step Four is to score the achievement and attitude 
measures used in your evaluation and to summarize these 
scores in a way that will enable you to answer your evaluation 
questions. In this step, you will also summarize and organize 
archival data such as attendance records, counseling refer- 
rals, or grade point average reports. 



As we move toward using performance assessments to measure 
curriculum goals and school quality, the scoring process becomes an 
important step in establishing both the meaning of a measure and its 
validity. With multiple choice or selected response measures, for 
instance, where for each item students choose from among given 
alternatives or where respondents indicate their response on a nu- 
merical scale, the score is simply the number of correct items or the 
frequency of respondents selecting a certain response choice. Scoring 
means simply "counting" the number of correct answers as deter- 
mined by a preestablished answer key. We know what the scores 
/ <>an on selected response and multiple choice measures by looking 
at the items and the knowledge/skill framework those items are 
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supposed to represent. For example, a score of 8 on a 10-item calculus 
quiz assessing the concept of integration suggests that the student 
understood the concept pretty well (as measured by this particular 
set of 10 items). Or a score on a questionnaire where 9C% of parent 
respondents selected "agree" or Very much agree" on the item a ir.y 
child is happy in school" means nearly all parents answering the 
questionnaire felt that their children were happy in school. The score 
meaning in both examples was determined prior to the administra- 
tion of the measure, at the time when the conceptual framework for 
the assessment was established (often called a content-process matrix 
in achievement test design) and when the items and their prespecified 
alternatives were written. 

For multiple choice or selected response measures, the test's 
f conceptual framework predetermines what "knowledge of calculus" 

or what "a good school" means, what the questions assessing them 
will be, and what the range of answer choices will be. Respondents 
react to given choices, and, in traditional multiple choice tests, their 
responses are either right or wrong, with nothing in between. 

Performance Although score meaning for performance assessments, essays, 
assessment scores, portfolios, projects, demonstrations, and the like is also a function of 

the kinds of questions asked and the expected responses, respondents 
are not given preestablished answers. As with selected response 
measures, performance assessments are grounded in a conceptual 
framework that defines the nature of the task or the nature of the 
questions students are asked to complete. It also defines the types or 
range of tasks that constitute good writing, scientific problem solving, 
social studies understanding, everyday math applications, and so 
on — but we can't anticipate in advance the entire range of possible 
answers. Thus the process of reviewing and judging answers to open- ? 1 
ended tasks (what we call "scoring" here) determines the score mean- 
ing as much as the initial tasks themselves. Scoring, for performance 
assessments, mean3 using a set of criteria to rate the quaiity of 
students' responses. 

The scoring criteria, often called rubrics, define the essential 
features of a quality response; they enable us to attach standardized 
meaning to students' performance, almost always along a continuum 
of quality. Although not all would agree that scoring criteria should 
be created before a performance test is administered, we strongly feel 
that, because such tests are being used at the school level to focus 
instruction and to assess school quality and student performance, 
fairness demands that expectations be public and known in advance. 
Teachers and students need to know what is expected of them, and 
therefore at least a draft set of criteria should exist prior to test 
administration. These criteria, scoring guidelines, or rubrics, repre- 
sent our best guesses (or best theory) of what reasonable continua of 
performance are, from novice or unskilled to highly expert, or what 
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constitutes different levels of understanding of a particular topic/ 
concept or field. 

Although our scoring criteria reflect our preconceptions of what 
the nature of a correct response is, much as the response alternatives 
in a good multiple choice test do, the difference is that, with perfor- 
mance assessment, the scoring criteria may be altered and adapted 
to reflect unanticipated or novel answers, and answers are rated on 
a continuum of quality, Thua scoring becomes more than a matter of 
counting and checking against an answer key. For performance as- 
sessments, scoring means judging thb quality of a response in com- 
parison with prespecified criteria (or a draft version of such criteria) 
and rating a response along a continuum (or, more often, continua) of 
possible responses. 

Outside versus The practical implication of this distinction between scoring a 
local scoring, selected response versus scoring a constructed response measure is 
that you will need to think of scoring differently for different types of 
instruments. For selected response instruments, hand scoring with 
an answer key or machine scoring either locally or by a test publisher 
is an easily managed and easily automated process that will not 
involve staff time in any substantial way. With performance assess- 
ments and with open-ended interview or questionnaire responses, 
however, the scoring process is also part of the process by which we 
understand what the scores and their underlying responses mean* To 
derive the greatest benefits from using these kinds of measures, you 
will want to involve in scoring the staff members who will be most 
involved with using the results. If, for example, you are having 
students compile science portfolios, you would want the science de- 
partment to develop (or adapt from existing models) scoring criteria 
that they agree are meaningful and suitable and then actually to score 
the portfolios. This way, members of your science department would 
be able to see the range of student performance and to understand 
how their particular students understand science as it is taught in 
your school. 

Although some test publishers offer off-site scoring services for 
writing samples, and no doubt will eventually do so for other types of 
performance assessments, if you use this service, you will lose the 
opportunity to involve your teachers in a review of both your student 
essays and the scoring guidelines. This review helps teachers come to 
a consensus about how student work should be judged, standardizes 
their judgments, and often carries over into instruction. We have seen 
this happen in high school English departments, after teat 3rs work 
together to score a set of grade-12 essays using the California Assess- 
ment Program rubrics. They apply those same rubrics in their own 
classrooms for peer review and for grading students. Similarly, we 
have seen teachers involved in the development and scoring of district 
writing assessments bring improved focus to their writing instruction 
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Different scores, 
different uses. 



Total score or 
subscale score? 



and use the scoring scheme to provide efficient feedback to students 
on their classroom writing. Over time, this produced improvement in 
students' writing performance. 

Prior io the scoring process, you will need to think about the kind 
of score that will be most useful in answering your evaluation ques- 
tions. There are a variety of different types of scores, and each has 
specific uses and limitations. Examples follow: 

• If you need to compare scores between different assessments 
that are supposed to measure the same things but with differ- 
ent numbers of items or different difficulty levels, you will need 
to use some type of derived score (percentile rank, standard 
score, scale score, normal curve equivalent, stanine) rather 
than the number-correct or raw score. 

• If you want to compare across different kinds of measures, you 
might use a derived score such as the percentage of students 
at the success or effectiveness level you designated in Step Two. 

• Questionnaire results typically are summarized as the percent- 
age of respondents choosing each alternative (or the percentage 
within a range) or as the mean 'tern score (an average over the 
item scale). 

• Writing samples and many performance assessments often are 
summarized as a mean (average) quality rating or as the percent- 
age of students scoring at or above some designated quality 
rating. 

Table 4.1 summarizes the most commonly used scores. 

In addition to the type of score you will use, you will be faced 
with choice in the level of detail you wish to report. Most standardized 
tests report total scores in each area assessed: typically, reading, 
language, and mathematics. Within these content areas, subtest 
scores for specific strands of skills also are provided, such as for 
vocabulary, math concepts, applications, and so on. The total score 
gives a general picture of performance and has the advantage of being 
more stable or consistent than subtest scores. Subtest scores, on the 
other hand, provide more diagnostic detail and help pinpoint areas of 
strength and weakness. 

Questionnaire's scores also are often reported in subscales — a set 
of questions measuring different aspects of the same topic, such as 
parent satisfaction with the school program, teacher expectations for 
student performance, or staff satisfaction with school policies. Re- 
sponses to individual items then help to identify areas of relative 
strength and weakness with regard to the subscale area assessed. 
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TABLE 4.1 Summary of Score Types by Instrument 

Nationally normed standardized tests 

Raw score: number correct 

Individual pupil percentile rank: percentage of students scoring at or 
below given score 

School/district percentile rank: percentage of schools/districts scoring 
at or below given score 

Stanine: section of the normal distribution into which score falls; 
1 = lowest and 9 = highest 

Normal curve equivalent: transformed percentile rank to create equal 
interval scale; used for Chapter 1 reporting 
Scale/standard score: transformed score on arbitrary scale such as 
SAT scores 

Selected resonse curriculum-related tests (including minimum 
competency) 

Raw score: number correct 

Percentage correct: number correct over total possible points 
Selected response questionnaires 

Frequency: actual number of people choosing a particular response al- 
ternative for a particular item 

Proportion/percentage: response frequencies converted into percent- 
ages for comparison purposes 

Performance tests 

Raw score: rating assigned to a paper/performance/portfolio by a judge 
based on a set of scoring criteria 

Archival data 

Frequencies: number of students in scoring categories such as "10-15 
absences" 

Proportions/percentages: frequencies converted into percentage? for 
comparison purposes 



The question of whether to use a total score or a subscale score 
becomes a little more complex when it comes to performance assess- 
ments. Although many writing assessments use only one holistic 
rubric and report a single overall score, you also will find numerous 
writing assessments that score essays on a number of analytic scales, 
using rubrics such as "organization," "content," "support," "style," 
"mechanics." You will need to decide whether you want a summary of 
performance or a profile that can provide more diagnostic, instruction- 
ally relevant information. 

Similar decisions need to be made in assessing complex perfor- 
mances that incorporate a variety of behaviors and content, such as 
projects or portfolios. An overall total score may suit your needs, 
representing an overall impression much as the holistic essay score 
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summarizes a variety of qualities, or you may use subscales scores, 
each representing different dimensions of performance. 

If you are tracking your school's progress over time, an overall 
impression or total score can simplify the task. When you are inter- 
ested program improvement, however, the profiles of student perfor- 
mance across different dimensions will be very useful. 



Selecting In this section, we suggest useful scoring procedures for the most 
Scoring commonly used evaluation measures beginning with the most tradi- 
Procedures tional, norm-referenced, standardized tests, and concluding with the 
rewest, performance and portfolio assessments. 



Published tests. 



For published standardized, norm-referenced multiple choice 
achievement tests and attitude surveys, you need only return the 
answer documents to the publisher, request scores and score reports, 
and await your results. We suggest that, in addition to percentile 
ranks, you request a raw score and a normal curve equivalent or 
standard score. 

The percentile ranks will be useiul for reporting results to parents 
and teachers, but the raw scores will help both to understand the 
absolute performance. For example, a parent once called, distraught 
that her 99th percentile student received an 85th percentile rank in 
grade-3 vocabulary. A look at the raw score reported with the ranking 
showed that, of 20 items on the test, the student missed only 3. The 
parent was relieved about her student's performance and enlightened 
about the "real" meaning of percentile ranks. You will need normal curve 
equivalent (NCE) scores for reporting test results for state and federally 
funded programs such as Chapter 1 or Title VII. These scores are 
useful for calculating averages, graphing results over time, and mak- 
ing comparisons between different groups. The standard score can be 
used for the same purposes as the NCE, but, because each test publisher 
calculates standard scores differently, these are not as useful for 
making comparisons between different tests. 

Your school district may have the equipment and the license for 
in-house scoring of published tests. In that case, you can ask the 
district data processing director to produce raw, percentile, and NCE 
scores. Although in-house scoring is quick and cheap, we have found 
that errors can creep into the test scoring and summarizing process. 
You will need to review the kinds of reports available and determine 
how summary scores are calculated. The most frequent mistakes 
we've seen are averaged percentile ranks (because percentile scores 
are not an equal interval scale, they cannot be averaged) and percen- 
tile ranks graphed on an equal-interval scale. We've also seen the 
mean score used to designate the 50th percentile rank instead of the 
median, which is not a serious error if your scores are normally 
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distributed but can be quite a distortion if you have a very high- or 
very low-achieving population. 

If some of your measures are scored by a publisher or at the 
district, you will be ordering score reports or summaries as well as 
specifying how scores are to be reported back. For program evaluation 
purposes, in addition to the usual class- and grade-level alpha lists of 
student results, you will want frequency distributions by class and 
grade level for each subtest. You may want your frequency distribu- 
tions to specify the number and percentage of students scoring in each 
stanine or quartile, depending upon whether you need to look closely 
at the percentage of students scoring in the extremes of a distribution. 
You will also want to order a "criterion-referenced" analysis that 
displays the percentage of students correctly answering the cluster 
of items measuring a single objective. An item analysis showing the 
percentage of students answering each item correctly can help iden- 
tify which items measure skills not taught or those that have difficult 
formats. Ideally, all of these reports can break down results by 
subgroups, such as boys and girls, limited and fluent English profi- 
cient, or students in the school for two years or more versus those new 
to the school. Score summaries by class and grade level and subgroup 
results by grade level form the "stufF of your evaluation. 

School-developed Every evaluation incorporates locally developed measures, many 
selected response of which use "Scan-tron" technology for summarizing results. The 
questionnaires, majority of items on most teacher, student, and parent questionnaires 
ask respondents to indicate their asponse on some numerical scale 
(e.g., a five-point Likert scale of agr*,e to disagree), either by circling 
their response or by marking their choice on a Scan-tron or optical- 
scan score sheet. You should know that special answer sheets can be 
purchased to enable you to score questionnaire items using your test- 
scoring Scan-tron equipment. With these special sheets, the scanner 
tallies the frequency of each answer choice, "a" to "e," for each item. 
When you push the button to get a "read out," instead of getting the 
number of students answering an item incorrectly (the usual Scan- 
tron read out), you get the number of respondents picking la, lb, 1c, 
and so on. If you copy this information and transfer it to a spread- 
sheet, you can easily calculate the percentage of people selecting each 
rating, 1 to 5, for each item, which then essentially becomes your 
questionnaire score. ( can-trons also supply Apple and IBM compati- 
ble software to fully automate the scoring of both tests and question- 
naires. The software allows you to produce the kinds of summary 
reports you will need to analyze your evaluation information. 

If your data processing department has optical-scanning equip- 
ment and the time to program software to score your selected response 
questionnaire, you will use optical-scan answer sheets. The sheets are 
read by the district computer and a report summarizing the number 
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School-developed 
open-ended 
questionnaire or 
interview questions. 



and percentage of people choosing each response for each question is 
generated by the software program. 

When you have only a small number of surveys, perhaps 30 or 
fewer, it is not difficult to summarize your information by hand. Set 
up a tally sheet with the survey question numbers and each response 
choice on the left-hand side. Tally the number of people selecting each 
response. Add up the tallies and convert the numbers into the per- 
centage of people choosing each response. The percentages can be 
calculated in either of two ways: the number of people selecting "a," 
"b," "c," and so on, divided by the total number of people responding 
to that question, or the number of people selecting a response alter- 
native divided by the total number answering the survey. You will 
find that not all respondents will answer all items so that it is often 
more accurate to calculate your response percentages using the totals 
for each question. 

Often, your most valuable feedback comes from respondent com- 
ments or answers to open-ended questions such as this one: "In what 
areas do you think we could do a better job?" These responses also can 
be summarized and tallied. First, read through the entire set of surveys 
to get an overall picture of the kinds of comments being made. Then, 
decide upon possible categories for these comments. For example, in 
response to the question, "In what areas do you think we could do a 
better job," common answers might include 



"Assign more homework." 

"I think the children need better playground supervision." 
"My student feels unsafe in the rest rooms." 
"I don't think my student is being challenged." 
"Why can't students earning a 'B' take prealgebra?" 

You can see three categories emerging in the above set of comments: 
campus safety, course expectations, and math assignment policy. You 
might have labeled these categories differently, but the point is that 
your categories serve as a starting point to tally and summarize com- 
ments. As you attempt to fit comments into the categories you have 
chosen, you will probably have to refine or redefine some of th^m. You 
should have a colleague read through the surveys and try to fit the 
comments into your categories as a check for consistency in labeling 
and tallying comments. Discuss any discrepancies and adjust your 
scoring procedure accordingly. The score for your open-ended ques- 
tions and comments will in fact be the sum of the tallies for each 
category emerging from the data. 

When you are summarizing open-ended questions, whether from 
an interview, questionnaire, or student work sample, be on the look- 
out for anecdotes or quotations that exemplify or characterize certain 
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Observations. 



School-developed 
writing samples 
and other 
performance 
measures. 



viewpoints or kinds of work. These examples will help illuminate your 
numerical summaries and clarify points of view encountered during 
your evaluation. 

Scoring observations will depend upon the observation strategy 
used. For an observation checklist or one that uses a structured, 
selected response format, simply tally and total tHe number of times 
each category was observed, much as was done vrith survey responses. 

The score for a scripted or narrative observation is obtained in 
the same way as the score for open-ended survey responses. You and 
a colleague can read through the observation scripts and identify 
descriptors or categories for the behaviors noted. Have someone tally 
recurring behaviors or classroom characteristics, and then check your 
categories and tallies by having another colleague go through the 
process. The observation score, like the score for open-ended survey 
or interview questions, will be the percentage of tallies falling into 
each category. 

Scoring writing samples entails three basic steps: (a) developing 
or adapting scoring criteria, (b) training scorers, and (c) holding a 
scoring session where you randomly assign papers to scorers and use 
procedures to assure reliability. Other types of performance assess- 
ment also require these same general steps. 

Although description of the details of each of these tasks is 
beyond the scope of this book, you might refer to a recent CRESST 
publication, Cognitively Sensitive Assessments of Student Writing in 
Content Areas (Baker, Aschbacher, Niemi, & Yamaguchi, 1991), for 
more specific guidelines. Some general information to keep in mind 
follows: 



• Definition of scoring criteria is a critical issue. Ideally, you will 
at least have a preliminary set of scoring criteria in mind when 
you draft your essay questions or performance tasks. Be aware 
that there are a number of models that can be adapted to your 
situation (and the number is growing rapidly as more states, 
districts, and others explore alternative student assessments). 

• Consider developing or adaptir rubrics that can be 
applicable in more than one r.»n*ent a/ea. For example, the 
CRESST rubrics developed by Baker et ai. to score content un- 
derstanding in social studies essays include "use of principles," 
"evidence of misconceptions," and "content quality." The rubrics 
also have been used to score science and math explanations. 

• The training and scoring session can be held on the same day. 
Involve teachers at the same grade level or of the same subject 
so that consensus on scoring guidelines will not be difficult to 
reach. 
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• You will need to decide whether an overall score will be as- 
signed to each piece or whether you will have subscores for 
different dimensions such as content, style, and support. 

• You will need to decide whether papers will be read by two or 
more readers (the recommended practice) or whether you will 
use some other procedures to assure the reliability of ratings. 
If you use two or more readers, plan to report an average score 
on each paper or to total the ratings. 

Portfolios and The scores from portfolios or from complex performances such as 
complex projects dance, drama, or art and student projects, can be extremely useful for 
and performances, program evaluation. The issues you consider are similar to those in 

scoring writing assessments. 

When scoring complex performances, you will have to consider 
whether one score is sufficient to describe student work or whether 
several unique scores are needed. For example, the Vermont mathe- 
matics portfolios require that portfolios include five to seven samples 
of student work and are scored on seven different dimensions, using 
a four-point scale of quality for each dimension, such as evidence of 
task understanding, reflection, analysis, language of mathematics, 
and clarity. The seven dimensions are summarized in two major 
scores, one for problem solving and one for communication. 

For program evaluation purposes, multiple scores such as those 
used in Vermont could be reported in a variety of ways: You could look 
at the average scores for each of the seven scoring categories. You 
could, as did Vermont, aggregate the categories into two major cate- 
gories, problem solving and communication, then average the scores 
for these two areas. Or you could calculate some kind of overall score. 



Score The scoring process is complete when you have score reports or 
Summaries and some type of score summary in hand for each instrument. For stan- 
Score Reports dardized tests, you can work with your sales representative to select 

the kinds of score reports most useful for answering your evaluation 
questions. Most often, these are some type of criterion-referenced 
report such as the percentage of students mastering each objective 
measu red by the test. Another often used score summary is the report 
of student score distributions on each test (frequency distributions). 

Your score reports for selected response portions of question- 
naires will show the entire distribution of responses for each item and 
may include the mean (average) response. If you are aggregating 
items into subscales, the report will show averages or distributions 
for each subscale. 

Reports for essays, projects, or peiformance tests can be handled 
in much the same way as tiv^e for more traditional tests. You can 
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report the percentage of students meeting prespecified standards 
represented by a certain rating, such as the percentage of students 
scoring "5" or better on a nine-point scale. You could report scores as 
the average rating achieved by students or you could report the 
percentage of students scoring at each point of the rating scale for 
each dimension. 

You may need to represent your score summaries in more than 
one way to clarify your evaluation questions. As discu sed m the next 
step, your analyses, interpretations, and reporting will be directly 
guided by these questions. For example, if you asked, "Are our stu- 
dents meeting district standards in writing," and your district hopes 
that all students can be rated as "competent" or higher on the district 
writing scale, you could summarize your scores as the percentage of 
students scoring below the "competent" score point and the percent- 
age scoring at or above that point. If, however, you are interested in 
knowing how to improve, you will want to know the percentage of 
students scoring at each level to see whether you need to work with 
a large population of low-scoring students or whether you need to 
improve in the higher ranges of the score scale. 

Worksheet 4.1 provides one type of a data summary. A blank 
working copy appears in Resource C. Once you have your scores sum- 
marized, you are ready to answer you evaluation questions, the task 
in Step Five. 

Suggested Most of the wisdom about scores and scoring is bur <ed in univer- 
Readings sity research or test publisher technical reports. Alth ugh an impor- 
tant topic to those having to manage the nuts-and-bolts aspects of 
scoring, scoring hardly merits a full-fledged book of its own. When it 
comes to developing scoring procedures and rubrics for performance 
tests, portfolios, and projects, be sure to avail yourself of presenta- 
tions at local and national conferences for current information in 4 his 
rapidly changing scene. Some recent releases we've found helpful 
follow. 

Baker, E. L., Aschbacher, P. R., Niemi, D., & Yamaguchi, E. (1991). 
Cognitively sensitive assessment of student writing in content 
areas. Los Anpeles: CRESST. 

This document is a virtual how-to manual for training teachers 
to use a research-based scoring rubric desi ncd to score high 
school social studies essays. It includes step-oy-step guidelines 
for preparing training materials, getting interrater agreement, 
and calculating reliability. Especially useful is the comprehen- 
sive scoring rubric that was developed using the best minds in 
content assessment in the country. The rubric can be adapted for 
local use. 
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Summarizing Data 


Instrument 


Score Selected 


Summary Method 


Questionnaire/Survey: 
Parent questionnaire items 
related to math program 
(Items 3, 6, 7, 9, 12) 


Percentage 
Frequencies 


Average percentages across categories "1-2" and M 4-5" 
Bar graph of percentage choosing "4-5" for each item 
Mean choice for each question (average of choices 1-5 on 
each item) 


Nationally Normed Test: 
ITBS Math Concepts and 
Problem Solving 


Percentile rank 
Standard score 


Line graph of median PRs for past 3 years by grade 
level and subtest 

Median percentile rank for each class and grade 
Distribution of percentage of students scoring in each 
stanine and in each quartile range 
Mean standard score 


Selected Response Curriculum- 
Related Tests: 
Competency test: Math 


Percentage-correct score 


Percentage scoring above the passing standard (65%) 
Bar graph of percentage passing math for past 5 years 


Performance Tests: 
State assessment: open- 
ended math item; Grade-8 
math portfolios 


Raw score (rating) 
Average raw score (rating) 


Frequency and percentage of students receiving each 
rating (1-6) 

Frequency and percentage of average ratings across 7 
papers (1-6 scale) 

Bar graphs comparing average ratings for girls versus boys 
Bar graphs comparing average ratings for LEP versus others 


Interviews (open-ended 
responses): 

Interviews with 3 students 

in each math class 


Frequency of answers 
classified in the same 
category 


List of student ' nments arranged from most to least 
frequent about aisfaction with math class 


Archival Information: 
Attendance; Course 
enrollment; Course grade 
distribution report 


Raw score: number of 
absences in math each 
period 

Rav» score: number enrolled 
in each math section 
Frequencies of grades A-F 
by course and section 


Average number of weekly absences for weeks 2, 5, 10, 16 

Line graphs of weekly absence rate for target weeks 

Average section enrollment by course for weeks 1, 3, 5, 20 

Line graph of course enrollment across time by course, 

grade 8, prealgebra, algebra 

Average u grade w (A-F) for each math course across 

sections 
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Finch, F. (Ed.). (1991). Educational performance assessment. Chi- 
cago: Riverside. 

Who said the appendix was a vestigial organ? The appendixes in 
this monograph produced by the publisher of the Iowa Tests of 
Basic Skills provide sample performance test exercises and scor- 
ing rubrics developed for the Arizona state assessment as well as 
sample leaders' and scorers' manuals. These documents make 
tangible procedures for scoring reading, writing, and mathemat- 
ics performance tasks. The chapter on prcMrrns in scoring per- 
formance tests by Steve Osterland, formerly of the Oakland 
(California) school district and now with the University of Mis- 
souri, alerts practitioners to the problems in scoring performance 
tests. In sum, there are many examples, and the book is practi - 
tioner friendly. 

Morris. L. L„ Fitz-Gibbon, C. T., & Lindheim, E. (1987). How to 
measure performance and use tests. Newbury Park, CA: Sage. 
Previously described as a resource for identifying tracking strat- 
egies, the chapter on validity and reliability of performance 
assessments should be read before you even think about setting 
up a scoring session. 
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Data 

Interpretation: 
A Dialogue 
About Meaning 



Step Five 

Analyze Information 
and Interpret Findings 



In this step, you will review score summaries and reports with 
your advisory group to answer each of your evaluation ques- 
tions. The focus during Step Five shifts from thinking about 
scores and summaries of particular measures back to the larger 
purpose of your evaluation: looking across those summaries to 
answer your specific questions. You will need Worksheet 4.1, sug- 
gested data summaries for each evaluation question, and your 
results to begin. 



Although the first image that comes to mind when we think about 
analyzii.^ data is a rather fearsome task involving numbers, scores, 
tables, graphs, and computer printouts, analysis is simply another 
term for a common human activity: looking for meaning, seeing 
relationships between parts and the whole. Your consolidated evalu- 
ation plan linking specific evaluation questions to particular measures 
provides a pathway for data analysis: You group score summaries and 
information by evaluation question, \, "n examine what answers 
emerge by looking at results relative to the standards you specified, 
by looking at trends over time, and/or by looking for relationships between 
program processes and outcomes for reporting. 
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Involve others in Data summaries and sophisticated analyses cannot direct an 
data interpretation, agenda for action. Data can raise issues for consideration among 

knowledgeable stakeholders. Data can supplement, but not supplant, 
our working knowledge. Data can stimulate thoughtful discussion 
about the nature of the challenges faced by the school and the progress 
it is making toward meeting those challenges. 

In short, the interpretation of evaluation data not only is a 
technical task but needs to be seen as part of the problem-solving, 
decision-making process ofxhe school. Discussion of systematically 
gathered and summarized information can be an effective manage- 
ment technique for building consensus about school goals and success 
if it involves key constituencies. These constituencies will help you 
generate the meanings and implications of the data you have col- 
lected, strengthening the support for change. The inclusion of teacher, 
parent, student, or community groups in reviewing data, further- 
more, permits a richer interpretation of the findings and can help 
provide explanations for results. 

A good strategy for traversing Step Five, then, is to provide your 
steering committee with the relevant data summaries — tables, graphs, 
survey summaries, and so on — bearing on each evaluation question. 
Ask members to review data related to each question and to speculate 
on the meaning of the results: What answers do they suggest? Have 
a member of the committee summarize the group responses. As you 
review the information, resolve discrepancies in interpretations, make 
a note of information that is incomplete or of places where you would 
like more information, and generate possible explanations suggested 
by the data or by your steering committee. 

The goal of data interpretation is to identify what courses of 
action would be most likely to help you meet your school goals. What 
changes in activities should be made? What kind of staff in-service is 
needed? What additional resources are needed? Should parent or 
community involvement be widened? For example, if our answer to 
the question, "How are we doing in math?" is "pretty well on the 
average but not so well for LEP students," then we will want to 
strengthen our program for these latter students. Perhaps the teach- 
ers on the advisory committee will suggest that they would like mor s 
help providing sheltered instruction to LEP students. Other sugges- 
tions could relate to a peer tutoring program and a bilingual parent 
evening where parents are shown how to help students in math using 
their native language. 

Certainly, the answers to all of your questions will not be found 
in tables of average test scores for different groups of students, graphs 
of responses comparing student-parent-teacher responses to similar 
questions on surveys, or notes from classroom observations. But 
hypotheses or suggestions derived from these data can initiate dis- 
cussion among the "experts" you have selected to help you review the 
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data. The discussions will validate some findings, will provide expla- 
nations for others, and also can help to uncover areas where findings 
are invalid. 

Although the evaluation questions for your school will be more 
specific than the ones we have used in this guide, the strategies for 
reviewing and interpreting data to answering certain kinds of ques- 
tions will be the same. Below, we suggest strategies for organizing 
and analyzing your data by question and then provide illustrations 
for needs assessment and formative evaluation qu stions. 



Organize Your 
Data by Question 



Remember the basic 
building blocks of 
data analysis. 



One of the most common mistakes made in data analysis is 
getting mired in the numbers and losing sight of the goal, which is to 
answer your evaluation questions. You easily can get so engrossed in 
the results of a single instrument, such as a standardized test or a 
teacher questionnaire, that attention shifts from answering a ques- 
tion to analyzing a particular set of scores. You spent a lot of time 
designing your evaluation so that you would have multiple measures 
to inform each question and to corroborate your findings: Stick to this 
focus! 

Therefore, while it may be easier and more efficient for you to 
prepare summaries instrument by instrument, resist the temptation. 
It will be much easier for your committee to synthesize information 
from multiple data sources if you only provide them with those results 
and score summaries related to a particular evaluation question. 
Table 5. 1 provides an example of a data display using multiple sources 
but edited to address only one evaluation question. 

Your data summaries by evaluation question will draw on the 
same score summaries that were presented in the last chapter. These 
included such summary statistics as means, medians, modes, and 
distributions (frequency of responses or number of students at par- 
ticular score points). Standard deviations and correlation coefficients, 
furthermore, are routine products of computerized score summaries 
and will be useful when answering questions about relationships or 
in comparing two or more score distributions. If you have forgotten 
some or most of the uses of these building blocks from your "Stat. 
101" class, the Suggested Readings at the end of this step provides 
several clearly written, practitioner-relevant sources you can use for 
references in preparing to analyze data. We also would like to add 
that this is one place you might want to call in an expert for help in 
selecting appropriate statistics and preparing clear data summaries. 
The expert can also review your questions and evaluation design and 
determine whether more complicated and sophisticated analyses such 
as significance testing are appropriate. 
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TABLE 5.1 Sample Data Organized by Evauation Question 



Question 1: How are our students doing in math at grade 8? 




State math test 


1991 


1990 




Concepts 
Problems 
Computation 


213 
245 
200 


200 
212 
200 




ITBS subtest 


1991 


1990* 


1989 


Concepts 
Problem Solving 


69 median PR 
73 median PR 


54 median PR 
60 median PR 


73 median PR 
63 median PR 



Norms change 

Parent Survey Responses to Statement: My child can complete math home- 
work without assistance (95% response rate) 

Percent 
SO 

40 
20 



1989 



991 



1990 



1989 



98S 


) 


199C 


1991 










n 





991 



Grade 6 



Grade 7 



Grade 8 



Comparison of Coastal Village Intermediate Math Scores with East Coast In- 
termediate and Gulf Coast Intermediate Schools: ITBS Problem Solving, 1991 



ITBS 

Student 

Scale 

250 



100 



x 



_L 



Coastal 



I eastern 



Gulf 



Remember that you In presenting data summaries to your committee, be aware that 
have several we all have different preferred ways of looking at information; some 
options in of us prefer numbers, but many of us can see things more clearly 
displaying data, through graphs and pictures. 

There are any number of formats for presenting your data. You 
can use tables to present frequency distributions or summary statistics 
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TABLE 5.2 Ideal Formats for Displaying Information 

• Bar graphs are best for static comparisons and comparing 
unlike kinds of information. Plain "vanilla" bar charts help 
people make more accurate comparisons than pie charts, 
stacked bar charts, or three dimensional bar charts. 

—Test reeults for one year 

—Comparison of reading and mathematics results 
—Percentage of people responding "Very Satisfied" to 

different questions on a survey 
—Comparison of reading scores at different schools 

• Line graphs are best for presenting trends. A single repre- 
sentation with multiple graph lines is better for representing 
multiple trends than several separate graphs. 

—Reading and mathematics scores from 1985 to 1988 
—Percentage of LEP students since 1984 for the state, 
district, and school 

—Trends in percentage of budget spent on certain cate- 
gories such as salaries, supplies, security 

• Pie charts are best for showing part-whole relationships. 

—Percentage of LEP, FEP, English-speaking students 
— Percentage of budget spent on various categories 
—Percentage of teachers holding emergency credentials 

• Pictographs provide rapid general impressions. 

—A graph composed of "happy faces" showing student 
attitude toward reading and mathematics 

—A bar graph of dollar signs showing how utility costs 
have climbed since 1980 

• Tables are best for providing exact numbers, but they are less 
likely to be read or unders tood than graphs. 



for several instruments in a compact format. You. can use frequency 
polygons (line graphs) to show trends over time. Bar graphs (histo- 
grams) are useful in comparing percentages and average scores on 
the same scale for different instruments, such as the math concepts 
and problem-solving tests. Table 5.2 presents some rules of thumb for 
displaying different kinds of data. If you incorporate these guidelines 
with those presented in Table 5.3, which summarizes the research on 
effective data display, you'll have no problem deciding how to prepare 
your data summaries. 
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TABLE 5.3 Guidelin es for Creating Effective Graphs and Tables 

General Guidelines 

• Do not write heading in all capitals. Use both upper and 

lower case. 

• Sentences worded negatively are stronger than those worded 
positively. These are best used to present information that 
is contrary to a reader's expectations. Because negatively 
worded sentences are more difficult to understand and are 
so emphatic, use them sparingly. 

• Questions at the beginning of textual explanations help read- 
ers focus on and remember important information. 

• Color coded bars and lines on graphs help readers compare 
information more easily than lines and bars in the same 
color. 

Bar Charts 

• Information should be labeled directly on the chart rather 
than through a "key" or "legend." 

• Horizontal bar charts leave more room for labels but may be 
more difficult to read. 

Line Graphs 

• Avoid clutter, both of lines and tic marks. 

• Choose the range of tic marks to include the entire range of 
the data; avoid scale breaks. 

• If scale breaks are necessary, do not connect numerical values 
on either side of a break. 

• Choose a scale so that data fill up as much of the region as 
possible. 

• Zero does not always have to be included in a scale. 

• Use a logarithmic scale when presenting multiplicative factors. 

Pictographs 

• Symbols should be self-explanatory and easily differentiated 
from one another. 

• Quantity is better represented by increasing the number 
rather than the size of symbols. 

Tables 

• Round numbers to no more than two significant digits. 

• Provide row and column averages. 

• Use columns to display the most important comparisons. 

• Order rows and columns by size of numbers rather than 
alphabetical order. 

• Set rows and columns compactly, so that the eye can make easy 
comparisons, rather than spacing them out across the page. 
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Use sensible data Multiple is a key concept throughout this guide. We tell you to 
analysis strategies, use multiple data sources to answer each question and advise you to 

use a variety of strategies to summarize the same data. The multiple 
exhortation becomes especially important during data analysis and 
interpretation because the use of multiple analysis strategies with 
multiple measures enables you to substantiate your findings. The five 
strategies we invoke automatically in addressing evaluation ques- 
tions are as follows: 

• Look at trends over time. 

• Compare results with a similar group or an appropriate norm 
group. 

• Compare results against a standard. 

• Monitor the performance of subgroups. 

• Look for relationships between process and outcomes. 

Once your have your data summaries and analyses organized so 
that they address each question, you are ready to close in on the 
answers about your school's success. 

Keep in mind, however, that the data analysis process is multidi- 
mensional. You will be answering questions, entertaining explanations, 
hypothesizing relationships among program components, and suggest- 
ing actions for program improvement all in the same session, which is 
why a linear sequence of steps such as those we have presented here 
can be a bit misleading. In fact, as you implement sensible evaluation 
strategies in your school, you will find that many of the tasks in oteps 
Four through Six overlap and occur simultaneously. 

A note on the As you examine scores over time or when you make comparisons 
significance of your between groups or between your school and others, you will want to 
findings, know: "Is the difference I see significant?" This concept of significance 
has at least two meanings: important and not simply a score fluctua- 
tion within the realm of chance. Before you can determine whether a 
score change (or even a difference in the percentage of responses to a 
survey item from one year to the next) is big enough to reflect 
significant (important) progress, you will need tests of statistical 
significance; these will tell you whether the change or difference you 
observe is one you could expect simply because of the amount of 
measurement error from one occasion to the next or whether the 
change is larger than the natural fluctuation associated with the 
measurement of human performance. Only if you find that your 
changes over time or differences between scores are statistically 
significant will you be able to say that they might reflect real educa- 
tional change. 

Under some circumstances, rarely encountered at the school level, 
the difference between two scores can be statistically significant 
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without reflecting any practical difference. Sample size is one factor 
that influences statistical analyses. When samples are large (more 
than 100), you don't need a very large difference between two scores 
for the difference to be labeled "statistically significant." For example, 
if 1,000 students take a standardized reading test in 1990 and get an 
average standard score of 283, and then the same group scores 288 in 
1991, the difference is likely to be found to be statistically significant. 
In examining the number of items answered correctly, however, we 
may find that, on the average in 1990, the number correct was 24 out 
of 30 and in 1991 it was 24.7 out of 30. To most of us, this difference 
has no real practical impact, nor would wc likely notice a marked 
difference in student classroom performance. 

Although you will not likely be running significance tests on your 
own data, there will be occasions where you may want an expert to 
determine whether differences you've found— over time or when com- 
paring two schools, scores, groups, or the like— reflect real differences 
in your program or represent expected score fluctuation due to mea- 
' surement error. What you should know is that, unless -hese differ- 
ences are statistically significant, you probably (not definitely) have 
no difference of any educational importance. On the other hand, when 
your results indicate that the difference over time or between two 
groups is statistically significant, you still aren't sure whether it is 
practically significant. You must exercise your own judgment by 
referring back to the actual performance being measured and what 
that score difference represents in terms of that performance before 
you can assert that "we have made real, practical, visible, measurable 
educational progress." 



Answering 
Needs 
Assessment 
Questions 

Look at findings A needs assessment looks across the range of programs operating 
in terms of in your school and identifies strengths and weaknesses. Because you 
prespecified criteria, must make some kind of judgment of success with a needs assess- 
ment, the most direct strategy for answering the "how are we doing" 
or "in what areas are we in need of improvement" questions is to 
measure your progress against a standard or criterion. If you have 
not already set standards against which you wish to measure your 
progress because you had no "feel" for what defensible progress might 
be or because you were using extant data summaries prepared for 
other purposes, now is the time to think about which standards 
represent success for your school. For norm-referenced, standardized 
tests of various types from multiple choice to the essay-based advanced 
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placement examinations, you could measure your success in some of 
the following ways: 

• Compare student test results with national or state norms. 

• Compare student test results with similar schools' performance. 

• Look at the percentage of students scoring above the 75th percen- 
tile, at the 50th percentile, and below the 25th percentile. 

• Look at the percentage of students scoring at predesignated 
"mastery" levels. 

• Look at the percentage attaining "passing" or attaining preset 
standards on state honors or regents examinations, advanced 
placement tests, o- international baccalaureate results. 

• Compare current dropout rates, average daily attendance, and 
percentage of unexcused absences with state goals. 

Table 5.4 displays data and compares results with prespecified 
standards for a middle school mathematics program. The table demon- 
strates how your method of setting standards will be determined by 
the kind of instrument you use. You can also see that, even though 
you may begin looking at your data to answer the "how are we doing" 
question, your analysis will lead you to consider simultaneously other 
issues such as "why did we not meet our goal?" and "how can we 
improve?" Once these questions have been raised in a discussion of 
findings, suggested actions for improving the program are not far 
behind. 

Look at trends A one-year snapshot of student performance does not provide a 
over time, large enough picture of your school. Did performance on key indica- 
tors such as attendance, dropouts, and grade point averages improve' 
Was this due to changes in the size and diversity of your student 
population or was it simply a one-year aberration? By looking at these 
same indicators for the past three to five years, you're in a better 
position to know whether the improvement is part of a trend. If this 
is your first year collecting information, you now have a baseline 
against which to compare future progress. Do not be too hasty in 
making judgments about school needs with only one year's wonh of 
information. If this is the second or third year of progress tracking 
see if the data reveal an improvement. 

Look for educationally significant changes in performance. How 
large a change must be to reflect a practical difference will depend 
upon the measure you use and what kinds of performance t? score 
change of a particular amount represents. Although some fluctuation 
in performance is expected each year simply due to differences in 
student population, is the general trend upward? Is your general 
tread downward, upward, or relatively stable and similar to schools 
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TABLE 5.4 Using Prespecified Standards to Interpret Data 



Instrument 



Standard Percentage Meeting Conclusion 



ITBS math 



State assessment 



Math competency 

Open-ended math 
question 



50% above 
50th PR 

Fewer in bottom 
quartile than 
last year 



Fewer fails than 
last year 

At least 50% 
scoring "accept- 
able" or above 



Grade-8 portfolios 



Average rating: 
4 out of 6 



Parent survey 



Course grade 
distribution 



61% above 50th 

Bottom quartile 
only 8% this year; 
15% last year; 
never lower than 
13% 

Fails: 35%; 
last year, 37% 

52% scored 
acceptable, but 
only 20% of the 
girls did 

20% were rated 
4 out of 6 or 
higher 



Course enrollment 



Above goal: 
success 

Met goal: 
success 



At least 80% 
register satisfac- 
tion (rate 4-5) 
with new program 



No more than 1% 
leave algebra for 
lower-level math 



At least 95% of 
students get "C" 
or higher 



About the same; 
goal not met 

Met goal overall 
but need to 
examine girls' 
performance 

Problem here; 
let's look more 
closely when we 
examine other 
data 

Met goal 
success in grades 
6, 8; let's look 
more closely at 
program in 
grade 7 

Algebra 1 transfer Problem here: 
rate 30% Standard unreal- 

istic? Kids not 
prepared? 
Teachers 
unrealistic? 

Close to goal: 
Did we set 
standard too 
high? Look at 
girls' 

performance 



90% said they 
were happy with 
the program at 
grades 6 and 8 
but only 70% at 
grade 7 



89% received "C n 
or higher, but 
only 75% of girls 
did 



representing similar programs and populations? Your answers to 
these questions will point to school-level strengths or weaknesses in 
particular areas. 

Table 5.1 displayed parent responses about homework over a 
three-year period. We can see that the percentage of parents responding 
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that their children could complete their mathematics homework with- 
out assistance increased steadily in grades 6 and 8 over the three 
years, while grade 7 has tne op; osite trend. Even though we have not 
tested for significant increases over time, if the parent response rate 
was nearly the same over the three years and the kinds of parents 
returning surveys were similar to those not completing the surveys, 
then we can suggest that the new program has possible benefits at 
two grade levels. Table 5.1 also displays test score data over time. We 
see that scores have gone up in math concepts and problems, but not 
in computation, on the state assessment and the ITBS for the past two 
years. This information, if corroborated, indicates successful program 
implementation. 

Before you decide that you have a certifiable success or a real 
problem, however, confirm your interpretation of needs by corrobo- 
rating tentative findings and by checking the performance of special 
student subpopulations. 

Corroborate This may be a baseline year, but if you have used multiple indica- 
tentatwe findings tors to look acros your program, you may be more confident about what 
with multiple you conclude to be your strengths or weaknesses. For example, if in the 
indicators, first year that you collect math portfolios you find that only 12% of the 
students evidenced a consistent ability to explain their problem-solving 
strategies clearly, you will want corroborating information before over- 
hauling this aspect of the math program. Do your classroom observa- 
tions suggest that students have few opportunities to explain their 
work? Are students given opportunities to discuss or write about their 
problem-solving processes in social studies, science, or language arts? 
Do teacher or parent questionnaire results also suggest a weakness in 
this area? If several data sources tend to point in the same direction, 
you have identified an area of success or need. 

When you organize multiple sources of data by evaluation ques- 
tions, as was done in Table 5.1, it is fairly simple to find corroborating 
evidence for your findings from any one analysis. Here we have two 
sets of standardized test scores showing the same upward trend in 
mathematics concepts and problem solving for the past two years. The 
ITBS score did drop between 1989 and 1990, but, as the test norms 
used by the school were updated, the new norms could account for 
some of the drop. The comparison of Coastal Village with similar 
schools on the ITBS Problem Solving subtest provides further evi- 
dence of program improvement. Coastal scores are higher than East- 
ern and comparable to Gulfs. More students are scoring in the middle 
and higher ranges of the distribution at Coastal than at Gulf, how- 
ever—a desirable outcome. 

Look at If you have improved and the improvement is corroborated, you're 
subpopulatwn still not ready to publish your success. The same caveat applies to any 
performance, initial findings of poor performance. The real purpose of evaluation 
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is to explain and find solutions for program improvement, not just 
describe your school. Examine findings by subpopulation. The goal of 
American education is to provide equal educational opportunity for 
all students regardless of race, language background, or handicapping 
condition. Therefore you'll want to be sure that you're being equally 
successful with all your students. 

Perhaps you have never "disaggregated" your data (looked at 
data by subgroups of students) before and don't know quite where to 
begin. Where are differences likely to occur? What should you be 
watching? State departments of education and large school districts 
generally monitor the performance of the following subpopulations: 

• boys versus girls in math and in reading-writing 

• limited versus redesignated versus native English proficient 
students 

• students from different socioeconomic backgrounds, often mea- 
sured by level of parent education or AFDC/free lunch participation 

• students of different ethnic backgrounds: African American, 
Hispanic, Asian, Pacific Islander, white, and so on 

• students new to the school versus those enrolled since grade 1 
or 6 or 9 depending upon grade-level configurations 

You may have different subgroups of interest that bear watching. Per- 
haps you need to look at students who have been retained one or more 
years (such as those participating in a prekindergarten or develop- 
mental kindergarten program). You may want to compare students 
who live in the neighborhood with those who are bussed to school. If 
you have some evidence that certain groups may be at risk for school 
failure, this is the time to examine their performance, however you 
define them. 

Table 5.5 presents student survey data by subgroup, boys versus 
girls, for the statement, "I usually understand how to do my math 
assignments." We see that a larger percentage of boys affirm this 
statement as they move through middle school while the percentage 
of girls saying that they understand their assignments plummets 
drastically from grade 6 to 8. This raises an issue: Despite overall 
improvements in mathematics performance, are all students benefit- 
ing from the math program? 

Your analysis of subpopulation needs should follow the same 
procedures as your school-level needs assessment: Look at trends over 
time; corroborate findings with multiple measures. In this way, you 
are "controlling" for random fluctuations in student population, stu- 
dent mobility, and less-than-perfect measures. 

Sample Worksheet 5.1 summarizes strategies for answering the 
"how are we doing" question (needs assessment) and provides exam- 
ples for answering program improvement questions as well. 
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TABLE 5.5 Sample Data for Answ ering Formative Evaluation Questions 

Question: Are we implementing the math program as planned? 
Teacher Interview Summaries (20 teachers) 



What new strategies used? 



What problems are we having? 



Cooperative learning; n = 20 
Student math projects; n = 14 
Math essays; n = 2 



Need more planning time; n = 18 
No time for writing esanys; n = 14 
Kids don't know math facts; n = 7 



Student Survey Summary (Grade h = 310; Grade 7 = 331; Grade 8 * 312) 
I usually understand how to do my math assignments and homewor' 



Grade 



Boys 



Girls 



6 
7 
8 



75% 
78% 
79% 



77% 
56% 
43% 



Grade-8 Mathematics Portfolio Summaries (random sample of 50; 10 per 
classroom) 

ITBS 



Class 


Papers 


Essay 


Problems 


Mean Rating 
(1 ■ 6) 


Concepts 
Mean SS 


1 


40 


2% 


10% 


2.3 


210 


2 


17 


30% 


45% 


4.7 


245 


3 


10 


20% 


50% 


4.2 


234 


4 


4 


0 


1% 


1.2 


195 


5 


12 


3% 


10% 


2.1 


200 


Average 


16.60 


11.00 


23.20 


3.10 


216.80 



Percentage of Time Spent in Cooperative Learning Groups by Grade 
Level Over Last 3 Years 



Percent 
Time 

HXl 

50 
25 
10 



Grade 6 




Grade 7 



Grade 8 
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I Sample Worksheet 5.1 

Data Analysis for Specific Evaluation Questions 


Quests : In what areas are we in need of improvement? 

Instrun>:n? Score Used Analysis 


CTBS 


Percentiles 
Percentile 
Derived score 

Derived score 


Table showing percentage of students scoring in each quartile range: 0-24; 25-49; 50-74; 75-99 

School-level percentile rank equivalent to the mean school standard score 

Bar graph showing mean standard scores for reading, math, and language for grades 3, 4, 

5. 6. (separate graphs for each crrade level) 
Line graph for reading showing mean score for last 3 years; graphs for math and language 


State assessment 


Percentile 
Percentile 
Derived score 
Derived score 


Table showing percentage of students scoring in each quartile range 
School rank in state 

Mean scores in reading for LEP, English, FEP students; for math; for language 
Line graph showing mean reading score for past 3 years, mean nnrth, mean language 


Competency 


Percentage 


Bar graph showing passing percentage for reading, writing, math 

Bar graph showing passing percentage age for special program students 


Parent survey 


Average 
percentage 
positive on 
program related 
questions 


Bar graph comparing percentage agreeing with positive statements about reading, math, 
language 


Question: How well are we implementing a process-based approach to writing? 

Instrument Score Used Analysis 


Observations 


Tally of 

techniques 

observed 


Table listing desirable writing strategies observed and frequency of each 


Teachers 


None 


List of successes and obstacles reported 



Question: Is student writing improving? 

Instrument Score Used 



Analysis 



Portfolios 
Student survey 



Teachers 



Essay rating 
Percentage 
saying "agree" 
or "strongly 
agree" on 
questions 2, 6 

None 



Bar graph showing percentage of students at each grade level rated U 4T or above 
Bar graph showing average percentage reporting writing is improving by grade level 



List of specific ways teachers report writing improved; teacher estimate of percentage of 
students still writing well below expectations 



Question: Do students feel positive about the writing program? 

Instrument Score Used Analysis 



Student survey 



Parent survey 



Percentage 
saying "agree* 
on questions 
9, 11 

Percentage 
saying "agree" 
on questions 5, 8 



Bar graph comparing average percentage of students with positive responses on questions 
9 L \d 11 with average percentage of parents reporting students are positive on parent 
questions 5 and 9 
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Answering In analyzing data to identify school strengths and weaknesses, 
Formative you look across a range of programs and develop a wide perspective; 
Evaluation when you ask, "How can we improve?" in formative evaluation, how- 
Questions ever, you want an in-depth view of specific programs. This view is 
concerned with "how" and "why." You want to know more than what 
your performance profile is; you are concerned about finding explana- 
tions for this performance and solutions, if necessary. Involve others in 
this search for explanations and maintain an ongoing dialogue about 
the relationship between school processes and outcomes. 

Check to see that One of the most common mistakes we make in looking at our 
you have a school programs is to assume that, just because we call something a 
"program" and that program, it actually exists. Your district has switched to "whole 
it is being language" teaching in grades K-6, has adopted materials, and all 
implemented, teachers have been to two whole language in-services. Your district 
mentor teacher has presented a workshop on integrating reading and 
writing, and all administrators have been shown how to observe a 
whole language classroom. Suppose your evaluation questions focus 
on the effectiveness of this new approach. If you are interested in 
improving this new program, first, be sure it exists. Have teachers 
renamed their actions but continued with the same practices, teach- 
ing literature as they would a basal? Are there inconsistencies in 
implementation by grade level? What problems are you having in 
implementing the program? Do you have enough materials? Do you 
all understand it in the same or a similar way? Do you feel comfortable 
with new strategies needed to make it work? The answers to these 
questions can be found, in part, by looking at data in some of the ways 
suggested below: 



• Look at observation and interview data. Listen to steering 
committee experiences. List the activities being implemented. 
Note problems encountered or successes attained. 

• Look at student, teacher, and parent survey data. Look for 
agreement as to program descriptions, activities. Look for re- 
ports of desired program outcomes such as increased writing, 
better attitude toward mathematics, and more willingness to 
set aside time to study. 



Program implementation data are included with the data sum- 
maries shown in Table 5.5. The teacher interview information shows 
an uneven use of new strategies, with cooperative learning being most 
widely used and writing about mathematics least implemented. Data 
from the student survey and a breakdown of the number of math 
essays in the grade-8 portfolios corroborate our supposition that 
teachers aren't having students write about mathematics. If this is a 
central component of our program, we have some evidence that it 
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hasn't been implemented as planned. The data also suggest that 
teachers want more in-service in this area and are not yet comfortable 
with this part of the program. 

Look for During the process of summarizing, you will begin to notice similar 
relationships patterns in different data sources. As you calculate absence patterns 
among school by grade level, sex, and educational placement, you may note that you 
context, process, find more absences among students in basic courses in grades 11 to 
and outcomes. 12. These relationships may help you pinpoint areas in need of 
improvement or provide explanations for your outcomes. Some of the 
most common relationships found in studies of schools include 

• socioeconomic status, length of time in school, language profi- 
ciency, and standardized test scores 

• attendance and grade point average 

• enrollment in academic courses and SAT, ACT, or achievement 
test scores 

• parent involvement rates, knowledge about school programs, 
and school achievement 

• student attitudes toward school, attendance, grades, and post- 
school educational plans 

In our previous example from Table 5.5, we find a possible relation- 
ship between program processes and outcomes. The two classes with 
the most writing assignments had the highest math concepts scores 
and portfolio ratings. Although this could be attributable to many 
factors, class composition and teacher style among them, it is a 
relationship that we may want to know more about and monitor in 
the coming year. 

Look for Many of the changes now taking place in our schools are those 
relationships based on theories of how students learn, how teachers improve pro- 
between the theory fessionally, and how curriculum should be organized to produce more 
underlying your meaningful instruction. These theories suggest relationships between 
action plans and changes we a-e making and certain outcomes we can expect as a 
predicted outcomes, result. For example, when you adopt a process-based approach to 

writing, you expect to find a greater quantity of writing and more 
students feeling that everyone can learn to write, not just the talented 
few. Use the predictions based on the theory behind the writing 
process to identify relationships between practice and outcomes at 
your school. Having asked students on a questionnaire how much 
writing they have done in the past few weeks and if they feel that 
"anyone can learn to write," you then might look at their answers in 
relation to average scores on student writing samples or teacher 
reports about the quality of student writing. The presence of a rela- 
tionship among these variables suggests that your theory is supported. 
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If you find no relationship, you might ask whether you have interpre- 
ted the theory correctly (is sheer quantity of writing related to im- 
provement or does the quality of writing instruction and/or type of 
writing have a bearing) or whether it's being implemented as intended. 
When your results contradict expectations, you have identified a 
program area worth investigating. 

Some theory-based relationships you might consider include 

• recency of teacher in-service activities and implementation of 
new techniques, 

• opportunities for teachers to try new techniques in a "safe" 
environment with peer support and inclination to implement 
an innovative strategy, 

• heterogeneous student grouping for academic activities and 
positive attitudes toward the subject, 

• use of calculators in mathematics and improved retention of 
number facts, and 

• integration of subjects such as reading, math, writing, social 
studies, and science for young children and improved reading 
comprehension and school motivation. 

In our earlier discussion of Table 5.5, we have noticed a possible 
relationship between the amount of writing and math test perfor- 
mance or portfolio rating. One reason this relationship stood out was 
that the theory underlying our program implementation activities 
suggests that, whe.i students have an opportunity to make their own 
meanings of numerical relationships, they have a better understand- 
ing of mathematics. Writing about math is certainly one way to 
engage students in thinking about numbers and how they "wo 
Thus the relationship that piqued our curiosity emerged from * 
theory of action. 

Look at subgroup Some of the same analyses you conducted to answer the "how are 
performance we doing" questions will be useful in ferreting out possible explana- 
for possible tions for program success or weakness. Strategies we have found 
explanations, useful include the following: 

• Look at achievement/results by subgroups to see if differential 
performance is partly responsible for results. Do we do well simply 
because the 20% of high-performing students raise our scores? 

• Look at changes in students population or mobility rates. 

• Look at the trend of variables related to student performance: 
attendance, length of time in the same school, socioeconomic 
status, language. Are increases or decreases in these variables 
over time coincident with achievement increases or decreases? 
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• Look at the curriculum offered different subgroups of students. 
Are differences in content/skills/pacing related to the perfor- 
mance of subgroups? 

• Look at student, parent, and teacher survey information. Does 
it suggest areas of weaknesses or reasons for performance? 

• Graph or calculate correlations between variables related to out- 
comes such as the relationship between attendance and grade 
point average or scores, or the relationship between years in 
the school at time of exiting the bilingual program. 

• Look at observation and interview data as well as personal anec- 
dotes to identify success stories and things that are working 
well. 

Use diagnostic Data can be used to point out areas for improvement or to "red 
subscalesfor flag" problems. They can be used to suggest possible reasons for 
program succtss or weakness. Their diagnostic potential, however, is limited 
improvement by the kind of measure used. For example, although they may provide 
information, indications of relative strength and weakness, the results of stan- 
dardized tests or any solely outcome- oriented performance measures 
will not tell you why students performed as they did (although 
analysis of subtest performance may suggest missing prerequisites if 
the underlying skills are believed to be hierarchical). Assessments, 
observations, and scoring guides that include process-oriented diag- 
nostic dimensions— based, for instance, on how students learn math, 
science, or social studies— however, can provide important insights 
about why students performed as they did. Similarly, selected re- 
sponse questionnaire and preference measures usually won't give you 
hints on how to improve but including a "comments" or "suggestions" 
section might. 

Descriptive and narrative information can help to identify why 
people feel as they do or to clarify program strengths or weaknesses. 
Subscale results also can help to pinpoint the latter. 

The subscale information of interest in our middle school math- 
ematics program example relates to the kinds of assignments in the 
student portfolios. We see that the percentage of papers devoted to 
problem solving, analysis, and explanation is low, even when the 
portfolios are hefty. This subscale analysis suggests that we could 
improve portfolio ratings (and maybe even test score performance) by 
increasing the number of challenging assignments in which students 
are asked to interact with mathematics in a creative, meaningful way. 

Sample Worksheet 5.1 maps the path for conducting an in-depth 
analysis of your program. It provides guidelines for answering both 
needs assessment (how are we doing?) and formative evaluation (how 
can we impicve?) questions. Answer these questions and you are 
ready for Step Six. 
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Suggested Entire careers are built upon analyzing data and writing statis- 
Readings tics books, which is probably why many published sources for this 
topic are virtually unfathomable by most educated Americans. WeVe 
found some that are fathomable and hope you will let us know your 
favorites. 

Fitz-Gibbon, C. T., & Morris, L. L. (1987). How to analyze data. 
Newbury Park, CA: Sage. 

This is a step-by-step manual for calculating correlation coeffi- 
cients, testing differences between groups, and calculating reli- 
ability. You don't need a statistics course to understand either 
why or how to do these procedures. Chapter 6 on the analysis of 
a body of data will organize the analysis procedure for even the 
most uninitiated. 

Jaeger, R. M. (1990). Statistics: A spectator sport (2nd ed.). Newbury 
Park, CA: Sage. 

If you're hiring a consultant and expect to get back sophisticated 
reports, complete with significance tests and correlation coeffi- 
cients, not to mention a regression analysis, this is the volume 
for you. All words, no numbers, no Greek letters. It explains why 
statistics are used, as well as when and how to interpret them. 

Morris, L. L., Fitz-Gibbon, C. T., & Lindheim, E. (1987). How to 
measure performance and use tests. Newbury Park, CA: Sage. 
This is another source with multiple uses. The final chapter on 
using performance test data provides guidelines for answering 
such commonly asked questions as these: How should students 
be placed in programs? How do they compare with a control/ 
comparison group? What are results like over time? Have pro- 
gram objectives been met? How do results relate to participant 
characteristics or program variations? This book is brief, concep- 
tual, and useful ibr performance data. 
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%0 Step Six 

Use Findings and Continue 
Program Monitor^ 



Overview 



Communicate 
Findings 

A bird's eye view 
of the terrain. 



The tasks in Step Six are few in number but complex in execu- 
tion. In this step, you make public your school's success and 
share your action plan for addressing any weaknesses. 



The central task in Step Six is to act. After gathering and summariz- 
ing data, you spent time in Step Five to reflect upon your findings 
with the others most involved in your school planning process. Now 
you will develop a plan for publicizing successes and strengthening 
your program where needed. In fact, Step Six is the pathway back to 
Step One, because what you decide to do in the coming year as a result 
of your evaluation provides the focus for the next year's evaluation 



You may draw upon a variety of methods for communicating your 
evaluation results. You may share your results informally through 
conversations with parents, teachers, or district staff or as just one 
agenda item at a faculty or school site advisory meeting. You may 
w sh to submit an informal article for the PTA newsletter or tack a 
note on the staff bulletin board. More formal settings could include a 
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presentation to the Board of Education, a report for an accreditation 
or school review committee, or an oral report at a parent meeting. 

This section describes some techniques for tailoring your reports 
to specific audiences. Plan to use a variety of strategies, formal, 
informal, oral and written, to influence your most important audi- 
ences (see Sample Worksheet 6.1). 

Take your audience Neither written nor oral presentations will necessarily report all 
into account, evaluation findings. Different audiences have different concerns and 
levels of sophistication when it comes to understanding data. Parent 
audiences want to know how their school "stacks up." They expect to 
hear of its successes but know that their school should be working to 
improve in some areas, even if it is doing well on the usual indicators. 
Our experience with parents in high-achieving schools is that they 
demand improvement and want specific evidence of instructional 
innovation despite high test scores and college placement rates. They 
are skeptical of reports containing only good news and are ever 
watchful of the whitewash. At the same time, these very concerned 
and involved parents will rally to improvement efforts, devoting 
buc 3ts of time and money for their schools. We also have found that 
parents in low-achieving schools have identical concerns with a some- 
what different twist: In addition to hearing about all of the weaknesses 
and all of the special programs designed to improve the school, 
parents are usually aware of special strengths—a particular teacher, 
an award-winning math club, a strong kindergarten program. It is 
important to highlight the individual and school accomplishments for 
the low achieving as well as the high. 

Teachers wan! to know both how their own students are doing 
and whether any instructional or curricular changes they are making 
have a payoff. Those who see no need to change need evidence that 
the change has positive results. Teachers who eagerly embrace a new 
program or strategy seek data to validate their beliefs and convince 
others. The biggest difficulty in sharing findings with teachers is that 
data don't always provide clear-cut answers or support for one view- 
point over another; this group then can become discouraged with the 
evaluation process and may remain unconvinced that any informa- 
tion other than teacher judgment is useful. 

The Board of Education, community at large, and, increasingly, 
the press are less directly involved but highly interested participants. 
These groups are less interested in the details of your findings but 
more interested in their credibility. These audiences are likely to 
raise technical issues such as "reliability," "significance," and "valid- 
ity" and are likely to want explanations and recommendations sup- 
ported with data. They view your successes with pride and widely 
pub icize them; therefore they need solid substantiation of results 
that say real change and/or real improvement has occurred. Neither 
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Sample Worksheet 6.1 

Communicate and Act on Findings 


Audiences 




Report Format 




Key Concern(s) 


Formal 


Informal 


Date 


Teachers: 
K-2 

Grade 3 
Grade 5 

All teachers 


Do children likp math? 

vstiiuA vii l J. iy \^ III dull. 

Do they understand 
"number"? 

Are children learning 
math facts as well as 
concepts? 

Do we still have the same 
number eligible for 
accelerated math at 
the intermediate level? 
Are we doing it right? 
Are we keeping up with 
tho other schools? 




0, H 


9/23 


Students: 


None this time 








Parents: 
K-2 

3-5 


Do kids like math? 
Are they keeping up? 
Are kids developing "skills"? 
Do they have homework? 
Can they solve problems? 


P--PTA 

P— Parent Education Night 


0 — School Advisory 
0— Open House 


10/3 
11/7 

1/14 
5/10 


School Board: 

. L 


Are the students maintaining 
or improving? What are you 
doing to improve program? 


T— Prior to meeting 
P— Regular meeting 

i 




9/12 
9/20 
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Sample Worksheet 6.1 

Continued 






Report Format 




Audiences 


Key Corcern(s) 


Formal 


Informal 


Date 


Community: None 










District/Other 
Administrators 


How's it gcir. 6 ,' What should 
we borrow? What kind of 
trouble are the teachers 
giving you about the new 
math program? 
Are you keeping up? 




0 — regular principal 
meetings 


8/21 
9/12 
10/3 


Other: 

Coastal Village Voice 


What new and interesting 
things in math are 
happening at your school. 


N — formal press release 




9/20 


Key for entering report format: 

H = Handouts: tables, sample items/performance tasks, score summaries, and so on 
N = Newsletter: school, district, PTA 

0 = Oral, informal report: faculty meetings, district meetings 

P = Presentation: formal, with invited audience, agenda, overheads, and so on 

T = Technical report: formal, written report following format of research report 
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Formal, written 
reports. 



Formal reports: 
School 
accountability 
report cards 



Formal reports: 
The media. 



they nor you want to be embarrassed by random reversals in next 
year's results. 

When you discuss areas needing improvement, a school board 
will most likely want to know the specifics of your action plan, with 
the focus on action. It is the group least likely to think that spending 
a year studying possible solutions is a reasonable first step to take in 
solving your school's dropout problem. 

Although you should plan to communicate your findings in a 
variety of forms, certainly, formal, written reports will be part of the 
process. Your most useful reports will be targeted to specific audiences 
and their concerns. Research conducted at the Center for Student 
Standards and Testing at UCLA on the use of information for school- 
based decision making pinpoints some useful guidelines for written 
reports targeted to site-level administrators and school boards. The 
guidelines summarized in Table 6.1 boil down to three rules of thumb: 

• Keep your report brief and simple. 

• Target your report to your audience's interests. 

• Use words and pictures as well as numbers to illustrate your 
points. 

Repeatedly throughout this guide, we have referred to school 
accountability report cards as an impetus for evaluation as well as a 
vehicle Tor highlighting school success. Although developing the first 
version may require enormous time and effort, once the report card 
categories are identified and defined, an appropriate format worked 
out, and the process automated , you will discover that your evaluation 
efforts make annual updates easy. We have included examples of two 
different school report cards meeting California accountability re- 
quirements, one from an elementary school and one from a middle 
school (see Resource E). Note how the format and level of specificity 
can vary while still meeting state reporting requirements. 

Some of your results will be reported in the newspapers or, for 
the less fortunate among us, on television. Your media debut most 
often occurs when and because the state has released test scores, not 
because you have called a press conference. Your local reporter will 
need lots of help understanding what scores mean, why they don't 
improve rapidly, and why they can fluctuate so much yet still not 
reflect "significant differences" in performance from one year to the 
next. We have found that, when we provide examples of test items, it 
is welcomed by the press. Most need quite a bit of education about 
measurement error and what a test score represents. Being able to 
bring to light other data bearing on school success also helps alleviate 
test score pressure. 
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TABLE 6.1 Hints for Cre ating Useful Reports 

Both school principals and school board members say that a useful report 

• is brief, 

• contains narrative or explanatory information 

• presents trends over time in graphic form to support conclusions, 
and 

• has a technical appendix for those interested in more detail. 
There are some differences in report content found useful by the two groups. 

• School-level administrators are most interested in the why and 
how of student outcomes. They seek information clarifying rela- 
tionships between school policies, curriculum and instruction, and 
school outcomes. 

• At the board level, not unexpectedly, the central concern is with 
the status of school performance and a description of progress 
rather than the nuts and bolts of planned improvements or reasons 
for performance. 

• When sharing information with school boards, there is a danger in 
presenting information at a level of detail and complexity that is 
not needed. 

Both audiences use and value qualitative information in decision making. 

• Written reports that include summaries of carefully conducted 
observations, interviews, or attitude surveys will provide powerful 
information for program improvement. 

• Teachers, principals, and others form conclusions about school 
effectiveness and hypotheses about how programs. succeed or fail 
based on impressions and experiences. 

• A few well-selected anecdotes also can be a powerful device for 
communicating the meaning of your findings. 

• Explanations of school results generated during the data interpre- 
tation process form yet another source of useful information for 
school-level decisions. 

• Qualitative data form an important part of the decision- making 
process; summaries of these data belong in written reports. 



Informal reports: Although formal, written reports for outside (nonschool) audi- 
Within the school ences probably will be prepared after ail data are in and interpreted, 

school-level constituencies will find information most useful and will 
be more willing to cooperate in data collection if feedback is fairly 
immediate. Teachers and other school staff form the closest, and per- 
haps most important, audience for reporting results. Most have been 
involved in one or more aspects of the school planning or improvement 
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effort as well as in data collection. Reporting to teachers and staff 
creates an occasion to clarify and amplify proposed explanations for 
findings and actions for the coming year. 

Remember that most people are most interested in the data that 
affect them personally Parents, for example, will be very interested 
in preliminary results of the parent survey. Teachers will be anxious 
to learn about how the student portfolios fared. We have a bulletin 
board posted outside our oflicr on which we post interesting feedback 
from teachers who are pilot testing materials so that the curriculum 
developers get more immediate feedback. Teachers also show great 
int/6i*6st) 

When reported promptly, even in rough form, data may help 
initiate some midcourse corrections. One intermediate school in the 
Los Angeles area found a surprising discrepancy between parent and 
student responses to a questionnaire sent out in January prior to 
developing their school plan for the next year. The question was this: 
"Students at this school respect others with different backgrounds." 
More than 90% of the parents and teachers responded that they 
agreed with this statement, but only 60% of the students registered 
the same feeling. Instead of waiting to address this issue in the next 
year's plan, the principal shared these preliminary results with the 
student council, who confirmed the student perceptions. The princi- 
pal then worked with teachers to set up a special team-building 
cross-grade homeroom that met twice a week for the last quarter of 
the year and helped students to know each other better. 

Informal reports: Another important reporting forum is the regularly scheduled 
The parent-staff school advisory committee meetings— whether the bilingual, special 
advisory committee, education, gifted, school improvement, Chapter 1, or consolidated 

program meetings. Advisory committees generally have a long agenda 
and little time; reports should answer the few, focused questions of 
special interest to the group: How are these special pjpulations 
doing? What do we plan to do to improve? How can we publicize and 
celebrate their successes? 

Informal reports: In When you send your standardized test scores home to parents, 
conjunction with you have another opportunity for sharing evaluation findings and 
other school implementing part of your school's action plan. Parents want to know 
business, how to help their children. Test score reporting provides an opportu- 
nity for your school to focus not only on individual scores but on how 
parents can become involved in the children's education. We have 
found that group meetings with parents that allow them to review 
their child's answer sheet and see actual items have raised parents' 
consciousness about testing. In 10 years of such meetings, a parent 
has never failed to make a public statement about how little the test 
reflects what the child has done in school that year. The parent 
forums present opportunities for sharing schoolwide patterns of test 
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Develop an 
Action Plan 



score performance and for discussing curricular changes that either 
account for this performance or may not be reflected in that perfor- 
mance. Although parents may not always be interested in attending 
an evening meeting to discuss new techniques used in the reading or 
mathematics program, they are extremely interested in their children's 
test scores. You can use this interest to segue to a discussion of the 
larger issues of their child's education. 

Excerpts of your findings, from how your school stacks up com- 
pared with others to unique school successes, can be reported in your 
regular PTA newsletters or at school functions such as Open House. 
High school principals appear particularly adept at integrating "good 
news" evaluation findings; one of our acquaintances announced the 
(very positive) results of the state assessment at high school gradua- 
tion! When reporting to your community, remember that many are 
unsophisticated consumers of data and have little knowledge of how 
schools operate. You should present your information in context and 
provide concrete examples both of the questions asked to gather the 
data (test items, survey questions, observation dimensions) and the 
action you plan to take based on the data. Even success requires 
action. Upon hearing "good news," parents and others ask, "What are 
you going to do to let others know how well we're doing?" 



In the process of analyzing and interpreting your findings, you 
no doubt at least touched on their action implications. Now is the time 
to set your prioriti's. 



Build upon your During your discussions, you identified those things you were 
strengths, doing well as well as the areas needing more attention. Do not, as 
many of us are wont, focus solely upon school weaknesses. If you are 
improving, make public your improvements to parents, district offi- 
ciate, and the larger community. If you are continuing to "hold your 
own" despite budget cuts, increasing class size, less aide time, or 
increasing enrollment, recognize your success and document it. If 
students and teachers perceive that something is working well, for 
example, the peer tutoring program, but parents either know nothing 
about it or discount it, spend some time with your key planners 
discussing how you can educate your public. 

But perhaps the most important consideration in reviewing school 
strengths and positive findings is how to build upon them in your 
action plan for the coming year. If you find that all English teachers 
have tried cooperative learning techniques in teaching writing and 
that most students like the group work associated with writing, think 
about how this technique could be used to strengthen other programs. 
Could your school consider using cooperative learning to improve thj 
poorly received, heterogeneously grouped mathematics classes? Could 
your English teachers serve as "coaches" for the mathematics teach- 
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ers who wish to try cooperative learning? Does your school have a 
really outstanding program in the arts, in interdisciplinary educa- 
tion, bilingual education, or multicultural education? Perhaps you 
should apply for special funding in one of these areas using the results 
of your evaluation to support your claims of program effectiveness. 

Have you found improvement in many areas? Perhaps this is the 
year to consider applying for a state or national distinguished school 
award or to join a network of schools dedicated to sharing program 
ideas in mathematics, technology, parent eduction, or other special 
programs. All schools have successes to share and improvement to 
publicize. Be sure to incorporate these successes into your plans for 
the next year. 

Address weaknesses. "Weakness" is a term with many meanings. To a high-achieving 

school with much community support, weakness is relative. In such 
idyllic circumstances, you could ask the question, "How do we keep 
on doing as well as we have in the past?" rather than the question, 
"How can we do better?" The action plan for such a school will use 
trends spotted in the evaluation data to anticipate potential future 
problems and attempt to solve them before they actually become 
problems. For example, a high-achieving school may have adopted a 
whole language reading-writing program but continue to test stu- 
dents with multiple choice tests. The evaluation findings may show 
that reading test scores remain high but language scores have dropped 
a bit. When whole language programs are implemented, student perfor- 
mance on discrete skills tests often falls, even though there are other 
program benefits. In this situation, the school could dedicate one 
section of its action plan to educating parents and the press about the 
mismatch between the standardized testing program and the new 
whole language program. It could also decide to use and report to the 
public results of more holistic assessments such as writing samples 
or reading-writing portfolios. A third action could be to teach children 
how to answer multiple choice tests and how the strategy is different 
than those used in reading and understanding a large chunk of prose. 

Some schools find that they are deluged with weaknesses from 
low scores to poor attendance to high student mobility to low parent 
involvement to a lack of bilingual instructors and little money for new 
supplies or in-service. The best strategy in this situation is this: Focus 
on doing a few things well. Here is where your evaluation findings 
will be really helpful. As you analyzed your data, you looked for 
relationships among school context factors or program strategies and 
outcomes. Identify the most powerful relationships you found. Will 
monitoring attendance improve the parents' perceptions of the school 
and academic achievement as well as garner community support? 
Will reorganizing the primary grade program and instituting devel- 
opmental, ungraded education address the readiness and retention 
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problems? Will adding counselors improve attendance and enroll- 
ment in academic courses and prevent dropouts? 



Continue 
Program 
Monitoring 

Document your 
plan. 



An action plan is simply your goals for the coming year and the 
strategies you will employ to meet those goals. The plan may be informal 
or it may already be a required component of a mandated school 
planning process, a formal plan that you file at the district each year. 
If you have special funding, your action plan is actually part of your 
school plan, perhaps Chapter 1 or Title VII goals. 

Your action plan also may be built into the staff evaluation 
process. You and the staff could incorporate at least one strategy to 
address a school need. Whether your action plan is a separate docu- 
ment, part of the school site plan, notations in the school calendar, or 
a set of goals to incorporate into the staff evaluation process, it is a 
commitment both to ongoing school improvement and to ongoing 
monitoring of your school's progress. Once you have devoted the time 
and resources to conduct a careful evaluation of school needs or of a 
particular program, you have set in motion a self-sustaining process 
for tracking your success. 

This brings us right back to Step One in the evaluation process, 
focusing your evaluation. The results of your first evaluation provide 
the focus for the second, and so the cycle is repeated. As with the 
example cited earlier in the development of a school report card, once 
the initial time has been spent exploring issues of concern to school 
constituencies, training an advisory group to think in terms of school 
context, processes, and outcomes, and gathering or developing a set 
of locally valid, reliable, and credible tracking strategies, the process 
becomes automated. 

We have completed the last step of sensible school-based evalu- 
ation. Although we have seen that acting on your findings and con- 
tinuing to monitor your programs lead us right back to Step One, you 
will be relieved to know that, for those not wanting to follow the 
logical sequence of events, which is to begin reading Step One again, 
we offer an exit to this guide that not only gets you out of the loop but 
moves you up to another view of tracking your success. 



Suggested As reporting your findings will most likely be one of your actions, 
Readings we highlight several sources for reporting information to various 
audiences. 
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Bain, J. G., & Herman, J. L. (1989). Improving opportunities for 
underachieving minority students: A planning guide for commu- 
nity action. Los Angeles: CRESST. 

If your evaluation results point to improving community involve- 
ment, this handbook provides a step-by-step approach for ac- 
complishing your plan. Full of worksheets and procedures, you'll 
iind templates and ideas for many kinds of action plans. 

Frechtling, J. A., & Myerberg, N. J. (1983). Reporting test scores to 
different audiences (ERIC/TM Report 85). Princeton, NJ: ERIC 
Clearinghouse on Tests, Measurement and Evaluation. 
These well-known evaluators from Montgomery County, Mary- 
land, have reviewed the testing reports from major school dis- 
tricts and developed guidelines for improving test reporting 
practices. This guide will help you present your successes clearly 
and keep you from making the most common mistakes in report- 
ing test scores. 

Herman, J. L. (1989). Political and practical issues in improving 
school boards' use of evaluation data. Los Angeles: CRESST. 
This paper summarizes field interviews with school board mem- 
bers. Unlike Freud, Herman did find an answer to her question, 
"What do school board members want?" The answer could save 
your career. 

Morris, L. L., Fitz-Gibbon, C. T., & Freeman, M. E. (1987). How to 
communicate evaluation findings. Newbury Park, CA: Sage. 
A whole book on reporting and it's about time! We like the parts 
dedicated to dealing with difficult audiences and the press (are 
they different?). 



12? 



Reprise and Revelation 



Evaluation Is 
Much Faster the 
Second Time 
Around 



You have just been through a six-step evaluation process adapted' 
especially for school people to help them look at their own schools. 
Although school-based evaluation may appear intimidating, remem- 
ber, you largely are just making explicit many of the processes you 
currently go through implicitly. You are building in some extra time 
and productively involving constituencies as you address routine 
school tasks, such as revising your school improvement or Chapter 1 
school plan, preparing accountability reports for the district or state, 
or getting ready for routine district and state program reviews! 
Sensible school-based evaluation requires that you approach these 
tasks with thoughtful questioning about what you plan to do and how 
you will know you have accomplished your plans. Sensible evaluation 
involves advance planning about the various kinds of information, 
most of which is already available at your school, that you will routinely 
gather throughout the course of a year to inform your school-level 
planning and decision making. 

The first time through, this process takes time: time for meetings 
to clarify school goals and identify concerns, time to find or develop 
progress tracking strategies, time to organize and review results, and 
time to brm;; people aboard in the decision-making process. Once you 
have identified appropriate instrument*, specified scoring strategies 
and reports, trained teachers and others in scoring strategies, figured 
out analysis strategies, and created report formats, the process be- 
comes "built in. w 

Watch how much easier and faster school-based evaluation be- 
comes the second time around. 



Ill 



ERIC 



1 



or 



112 



TRACKING YOUR SCHOOL'S SUCCESS 



Step One. Focus the 
evaluation. 



This time, your evaluation questions will emerge naturally from 
your analyses and interpretations of your first year's effort. If, during 
the first year, you conducted a needs assessment to profile your school 
strengths and weaknesses, the areas that you determined needed atten- 
tion and the plans you identified for addressing them would be the focus 
of the second-year evaluation. For example, you may have found that 
you didn't meet all the standards you set in areas such as dropout rate, 
mathematics achievement, parent satisfaction with the mathematics 
program, and student perceptions of intergroup toleration and cooper- 
ation. On that basis, you formulated some improvement plans. These 
improvement plans subsequently could provide a focus for a year-two 
formative evaluation effort in which you ask the question, "How effective 
are the planned improvements?" If you are in the enviable position of 
having met all your standards, you may wish to look at the performance 
of potentially at-risk subgroups of students to identify their special 
needs or to look at state mandates regarding the implementation of a 
new assessment program or a new mathematics program to provide a 
focus for school planning and evaluation. Regardless of the specific 
circumstances in your school, your year-two evaluation questions will 
emerge naturally as you interpret year-one findings and develop an 
action plan. Thus the time you spent during the first year's evaluation 
in bringing together important constituencies and initiating discussions 
about possible evaluation questions will not be necessary this second 
time around. These tasks will have been completed during the develop- 
ment of your action plan. 

The measures you adapted, selected, or developed during year 
one can be used in the second year. You may have to change a few 
questions on the parent survey or tighten up the scoring rubrics tor 
the science demonstrations, but your year-one measures can remain 
essentially intact for year two. If you should decide to look at com- 
pletely different areas, ask yourself if the "student attitudes toward 
reading" survey can be adapted to measuring your new focus, such as 
attitudes toward science. Or can your history portfolios serve as a 
model for developing a mathematics portfolio? Can the standardized 
test you used to look at school needs during year one serve a more 
diagnostic function in year two if you order different kinds of scores 
such as item analyses or reports of average grade-level performance 
by objectives? If your school records were less than useful during year 
one, can you change the record keeping system to make them useful 
now? 

Your data collection schedule, if successful, can serve as a model 

Manage instrument for year two. Your experience will help you adjust dates if some of 
development and your time 1 nes were too short. Remember, instrument development 
data collection, will not tako up quite so much time this year, as you most likely will 
be adapting year-one measures whenever possible. You also may feel 



Step Two. Identify 
tracking strategies. 



Step Three. 
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that conducting a written parent survey is not necessary every year 
You could consider sampling over a two- or three-year cycle and 
getting input from your school advisory committee and PTA during 
the intervening years. 

You now have trained data collectors, perhaps even parent vol- 
unteers, and teachers proficient at scoring some types of performance 
assessments. You will save time (and money) if you can use some of 
these people during the second year. We have found that, once we 
have a data collection schedule that accommodates school holidays, 
Open House, and parent conference days, scheduling for subsequent 
years will remain the same with only the minor adjustment of chang- 
ing exact calendar dates. Your data collection plan simply can be part 
of the school master calendar. 

Step Four Scoring You made your major scoring decisions and specified score re- 
and summarizing ports the first year. You also provided training for ^ 
data, portfolios, and the like. As long as these trained staff are still at your 
school you need only plan a refresher prior to scoring. You also can 
order or use many of the same score reports. Scoring becomes a more 
automated process. With archival data, scoring can be simplified if 
you have computerized records, a topic to which we return later. In 
any case, the time spent in scoring decisions is significantly reduced 
if you use the same or similar measures a second time. 

Step Five Analyze Use the data presentation techniques that worked best with your 
information and advisory committee as templates for presenting year-two data. Again, 
interpret findings, having computer templates will speed up the process, but even in 

their absence, the process of thinking through your data analyses the 
second and subsequent times around will be much faster and easier. 
You no longer have to decide how to organize information and try out 
different display strategies until you find the most useful ones. 

One place you will not want to stint on time, however, will be the 
actual process of reviewing your findings with involved constituencies. 
The heart of the evaluation process is interpretation and judgment. 
What happened? Was it good or bad? Why did it occur? How do we do 
better? These issues, though the same every year, have no fast answers 
and are always more clearly understood from multiple perspectives. 
"Protecting" teachers from the sometimes contentious discussion of 
school results has the unfortunate side effect of losing an important 
perspective in school improvement. Although not everyone is elated at 
the prospect of looking at tables and graphs or model portfolios at a 3:15 
p.m. faculty meeting, you just never know what kinds of observations 
and explanations about school performance will be uttered from the 
back row. "Outlier" interpretations and viewpoints may stimulate un- 
conventional solutions to heretofore intractable problems. 

Although the amount of time needed to look at findings may not 
be less the second time around, people will be more comfortable using 
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data and will need less time to understand tables and graphs. You 
may find some who look forward to the process because, for the first 
time, they see "hard evidence" to back up what they know to be 
occurring in the classroom. This validation of teacher judgment holds 
a central position in sensible school-based evaluation. 

Step Six. Use The reporting formats that worked well during year one become 

continue program cards and can include presentations at parent meetings or board or" 
monitoring. sentaUons Most likely your school board is too busy (&y S 
budget cuts) to hear from you every year, which cuts down the time 
needed to prepare a formal written technical evaluation report. On the 

2¥ ? ai : ent , S ^ C ° me J° expect and l00k forward to an annual 
testing" or school evaluation" report from which they can get more 

iSTr^ff 11 " Children 8 te8ting sch001 expectations 

and standards, and how your particular curriculum and instructional 

I n ting thdr Children ' 8 needs ' We ha ve found that we 

™ ! hy - S *? me 8X1X1X181 parent testin * meetings, but, when 
we fed the monster information about th ■ tests and the curriculum it 
was a beast that toile d f and supportea ug (And J*JM* 

neads was pretty simple the second time around.) 

p fl «ilv imiIarly ' ^ Pr ° CeSS ° f monitorin g your action plan is more 
easily incorporated into your daily activities the second time around 
Because you and your staff know the year's focus and have generated 

to dX5 7 y0l \ Want T T Gd * nearly evei * one has an <**na raised 
inf ° rmatl0n . relat ed to your school's plan. And, being think- 
ing human beings, for every piece of information, someone has an 
explanation or idea that can be incorporated into the plan for next 
IZ'J jr fSE Plan 18 stimulated ^ « mandate to develop a 

the ^ST^ lty K 6P0rt ^ GaCh year ' program monitoring in 
the report card areas becomes something all staff attend to in some 

fashion even if it's just to grumble about it). As for developing your 

action plan (i.e your school goals) for the following year; you will 

fin^H fmd thG f ° CUS Gmergin * from your dis 

findings during Step Five, which brings us back to Step One and 

focusing your subsequent evaluation. Pretty painless, wasn't it? 

,! 8 ^oaT, 0 ^!! I 6 k • At tHiS P ° int in 0Ur discus sion of the evaluation process we're 
to Automate beginning to feel like the introductory statistics professor whc^ insists 
you calculate correlation coefficients by hand so you will understand 

by hand. If you have noticed something familiar in our reprise you 
wil have observed that the words automate and template S 
text. These two words bring to mind our friend the computer Should 
you have access to a personal computer, it is simple to L.mate the 
more tedious and labor-intensive aspects of the evaluation pro est 
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What to automate. 



Focus the 
evaluation: 



Identify tracking 
strategies: 



Plan data collection: 



Score and 
summarize data: 



Analyze and 
interpret data: 



Act on findings 
and monitor your 
action plan: 



Prior to purchasing equipment or software or to dedicating any 
part of a computer system to evaluation functions, you should specify 
what you need to automate. What tasks you need performed will 
determine the equipment and software required. The following tasks 
are naturals for automation. 

• date base with the names and addresses of school advisory 
groups for producing mailing labels (if your school is completely 
automated, use electronic mail for meeting notifications and 
reminders) 

• word processing files containing meeting minutes and brain* 
storming suggestions for evaluation focus 

• text data base with items for questionnaires, observations 
interviews, curriculum-related tests, performance tests for cre- 
ating new forms of instruments 

• if you have no text data base (such as Notebook III), use your 
word processing program and revise measures with cut-and- 
paste procedures 

• automate your calendar and create a "tickler" file to remind you 
of tasks to be done each day/week 

• purchase project-management software 

• data base for scores or responses that will create list reports 
and summaries and calculate means 

• graphing program to graph results 

• data base that "talks to" a spreadsheet or simple statistical 
package and allows you to calculate statistics using performance 
test ratings, test scores, or proportions of response to question- 
naire items 

• student data base and analysis/reporting templates to track 
performance and other indicators over time 

• templates for your school plan ba?;ed on state or program 
(Chapter 1, Title VII, and so on) requirements 

• template for your school accountability report card 

• templates for your PTA newsletter or press releases 

• templates for parent presentation overheads 

• templates for district school board reports 

• templates for teacher evaluations observation notes (useful for 
personnel and program evaluation) 
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Hardware You will, of course, need a computer and some software that 
considerations, allows you to store information, sort it in several ways, create sum- 
mary reports of that information, and maybe even graph it. this 
means that you will need a computer with at least 20 MB of hard disk 
storage to hold the newer user-friendly programs that take up so 
much space, such as Windows or the Mac graphics-based programs, 
as well as your data. If you have a large school (say 1,000 students or 
more), you will appreciate having speedy disk access. As for computer 
peripherals, you will need a printer and a monitor. If your school has 
a well-heeled patron, ask for a laser printer (wonderful for producing 
accountability report cards), a page display monitor, a scanner for 
reading text into the computer, and a data scanner. Plan to get expert 
advice on what to buy and how to set up necessary programs and 
programming routines. 

' You should know that Scan-tron has a test analysis package that 
you can install on a microcomputer that you hook up to a Scan-tron 
scoring machine. The package allows you to create score reports for 
any kind of measure that has selected responses, including question- 
naires and tests that allow more than one correct answer. You may 
wish to investigate this option if your school is a large one and has to 
score and report minimum proficiency tests on a routine basis. The 
software is especially useful for analyzing survey data because it 
prints out the number and percentage of people responding to each 
answer choice on each survey question as well as item means. Calcu- 
lating this information by hand very quickly becomes extremely 
tedious. 



Software 
possibilities. 



You have several software options for automating evaluation 
functions. You could purchase a school-based evaluation system sim- 
ilar to the one devised for an elementary school by Bill Cooley at 
LRDC in Pittsburgh, Pennsylvania (Bank & Williams, 1985), de- 
scribed in the bibliography for this chapter. You could purchase a 
software system devised specifically for schools that has scheduling, 
attendance, and testing modules that summarize data useful for 
evaluation. Another option in to use your existing software. At a 
minimum, you will need the following capabilities: 

• Word processing: templates for data display and tables; tem- 
plates for school report cards; templates for reports and news- 
letter articles; "item bank" of questionnaires, observations, and 
performance tasks and rubrics 

• Spreadsheet: for calculating averages, standard deviations, 
percentages, and other summary statistics and for creating 
simple data displays with bar or line graphs 

• Data base (sometimes incorpoiated into the spreadsheet or 
word processing program): for storing attendance data, test 
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scores, and any codable/quantifiable information that you may 
want to sort in different ways and for creating different kinds 
of reports 

Many of us already have "integrated" programs that perform two or 
more of the above functions. Microsoft Works, Appleworks, Q&A, Excel, 
or Word Perfect are examples. We would caution you against using 
full-tilt relational, stand-alone, data base systems that require a pro- 
grammer to enter data and develop reports or any program that you 
are unable to vse after initial training. If you are shopping for data 
management software, tear out the questions listed below and refer 
to them surreptitiously as you query the salespeople: 

• "What kind of training do I need to use this software?" 

• "How many cases [individual student records] will it handle?" 

• "How long will it take to sort 1,000 records? 2,000? [or number 
of your choice?]" 

• "How long will it take to run lists or labels for 1,000 [or number 
of your choice] students?" 

• "Show me some reports produced by your system that are 
similar to these [produce your score reports and data summa- 
ries from year one— that's why we had you go through the 
process first]. Can your system do this? If so, what do I need to 
do to produce this report?" 

• "What kind of free technical support is available?" 

• "How compatible are files from this software with [name your 
word processing and spreadsheet programs]?" 

If you have software capable of calculating means, medians, modes, 
percentages, and creating bar and line graphs, you have the basic 
tools for analyzing evaluation data. There may be times you need to 
call in an expert for more sophisticated statistical analysis procedures 
(or you wish to do them yourself, perhaps as a graduate school assign- 
ment). For these occasions, you will need to have at least passing 
knowledge of what kinds of files your data management program pro- 
duces (you will need an ASCII file for most applications) and what 
kinds of files the statistical software package will import. 

As you consider automating all or part of the evaluation process, the 
natural question arises: "When do I need to bring in help?" Here is the 
second revelation of this chapter: You may need to call in a consultant. 



Getting by With a 
Little Help From 
Your Friends 



It could be that you have no intention of conducting an evaluation 
on your own. You are sitting on big bucks and all you want to do is to 
hire someone to get the job done. Or it could be that, after reading 
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this guide, you have identified areas where you feel overwhelmed or 
where bringing in an expert would save you time or money. Or you 
perhaps have access to district, county, or state evaluation consul- 
tants who can't shepherd you through the entire process but who 
could, if used at key times, make the process move more smoothly. 

You should consider bringing in a consultant, at least the first 
time around to work with you every step of the way or just at strategic 
points (depending on your resources). For either strategy, we recom- 
mend you find someone who understands schools, preferably your 
kind of school, and who has expertise in the areas where you feel least 
confident. 

• Bring in someone who is technically experienced and who 
understands schools. This does not mean a graduate student in 
statistics or a teacher working on an advanced degree in ad- 
ministration. Look for someone with expertise in evaluation 
and assessment as well as experienced in schools (with grade 
levels similar to those in your school). 

• Ask for a work sample; look at the writing and the presentation. 
Can you understand what is being written? Do the conclusions 
follow both from the evidence and from what you know about 
schools? Is the text reader friendly? Are the methods carefully 
described? Are results supported by data rather than intuition? 

• Interview the candidates and look to see if they understand 
what your school is attempting to do and trying to accomplish. 
Look for an ability to relate to teachers and others on staff. Be 
sure that the candidate listens to your concerns and is willing 
to come up with new or at least adapt his or her strategies to 
your school. Beware of the consultant who uses the same design 
or the same techniques for every situation. 

• Review the consultant's resume for experience in doing the 
kinds of things that will be most difficult for your staff (e.g., 
considering and devising a variety of assessment instruments, 
figuring out scoring schemes, running reliability analyses, spec- 
ifying and conducting appropriate analyses). 

Whether you have a consultant work with you every step of the way 
or use one at key points, the following are the most crucial areas and 
where you'll want to spend your time and money. 

Focusing. If this is your first attempt at school-level planning and evalua- 
tion, and you are facing a somewhat hostile staff, you may wish to 
bring in a neutral, friendly consultant to set up an open and accepting 
environment for focusing activities. The expert can help in getting 
people to clarify their evaluation concerns, without them thinking 
you're looking for particular answers, and can help you select the 
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kinds of questions that will be most informative for your particular 
situation. 



Tracking strategies. 



Instrument 
development and 
data collection. 



Scoring and 
summarizing. 



Analyze and 
interpret findings. 



At this stage, you'll want a consultant who already is familiar 
with a wide range of published measures, with measures and strate- 
gies being used successfully in schools like yours, and with the entire 
instrument development process. Especially important will be select- 
ing someone with actual experience using the kinds of measures you 
will be using. 

If you are doing any kind of performance assessment, you prob- 
ably need a consultant, if only to train your "lead" teachers, for 
designing or adapting performance tasks, for developing scoring ru- 
brics with staff, for training scorers, and for assuring that credible 
levels of scoring reliability are met. 

As discussed in the previous step, instrument development can 
be a quagmire for the novice. As you wish your results to be credible 
and defensible, you should seriously consider getting help, or at least 
expert review, if you plan to develop your own questionnaires, obser- 
vation scales, interview questions, or curriculum-based tests and 
performance assessments. Data collection, on the other hand, is labor 
intensive. You will probably not want to spend your "expert" consul- 
tant dollars on a person to copy data from archives, to walk into 
classrooms and administer standardized tests, or to make follow-up 
phone calls to parents who haven't returned their questionnaires. You 
may want to have an expert assist with classroom observations or 
staff interviews to provide another, more neutral, perspective on your 
findings. 

For published tests, this process is done by the test-scoring service. 
For performance tests, bringing in an expert, at least for the first 
time, can provide a wonderful staff development opportunity. Devel- 
oping scoring criteria, conducting rating sessions, and establishing 
rater reliability require experience and expertise worth the expendi- 
ture. On the other hand, you probably don't need any special advice 
at all to score and summarize questionnaire data, unless it's to set up 
automated computer programs. If you have videotapes of classrooms, 
extensive scripts of classroom observations, or other kinds of narrative 
data, you may want help in setting up scoring categories and creating 
a procedure for rater reliability. 

As with instrument development, here is an area worth some 
expenditure. The expert can determine what kinds of statistics should 
be used to summarize data, create score reports, and synthesize data 
from several sources. The expert can help your advisory committee 
look at the tables, graphs, model portfolios, and other information 
needed to answer your evaluation questions and can demystify the 
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data interpretation process. You will find you get your money's worth 
if you create a teaching relationship between your consultant and the 
school; get your expert to train you in how to analyze and interpret 
findings. 

Act on findings, It's not likely that anyone but school staff can do this job. You 
continue to monitor may wish to use an expert, however, to write any formal, technical 
your program, evaluation reports that will be subject to public, perhaps hostile, 
scrutiny and that will have to stand up under tough questioning. You 
might have an expert share the podium at a parent education evening 
or present findings to staff (we've found that high school principals 
don't mind having a little help from their friends once in a while). An 
outside consultant could even set up your accountability report card 
template. Beyond these areas, communicating findings and acting 
upon them remains the province of the school. 

As you have accompanied us through the steps in tracking your 
success, we hope you have been able to discover for your own partic- 
ular school the benefits of evaluation activities. We view evaluation 
as a decision-making process, something we all do intuitively, though 
not always systematically. Sensible school-based evaluation simply 
systematizes a very human activity: Ask a question, get information, 
make a judgment/decision. We hope, too, that we have communicated 
the importance of focusing on your success or on strategies of attain- 
ing success through program improvement rather than on focusing 
on "remediation" and "explanation." 

We firmly believe that school improvement is built upon the 
successes we have already attained or the standards for success we 
hold. In tracking, or hunting down, these successes, we engage in a 
process that helps us identify important relationships among school 
contexts, processes, and outcomes. When we do identify these rela- 
tionships and can replicate them, we have given ourselves powerful 
tools for school improvement. When we find out, for example, that, 
when teachers allow students to collaborate on assignments in heter- 
ogeneously grouped mathematics classrooms, both the academic per- 
formance and the self-concept of students of all abilities improve, we 
have powerful evidence of school success as well as a pathway for 
reaching our success in other programs. 

We hope that you will find reading this guide as useful as we have 
in writing it for clarifying our thoughts about evaluation, especially as 
conducted by school staff in a local setting to address immediate needs. 
We also hope that we leave you better informed consumers of evaluation 
results and better able to hire the kind of expertise that will help you 
make your school the success you know it can be. And, because we live 
in California, where the bumper sticker mentality still prevails, our 
parting words are these: Keep on Tracking! 
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Suaaested We have three major suggestions here: (a) As a reprise of ^the 
pSadfnas evaluation process, read the Evaluated Handbook referred o in Step 
Readings evamauo p mentioned in the Introduction (b) If you 

wish t automate, we have a few suggestions below, but the advice 1S 
stiU know as much as you can about what you want prior ,o under- 
tak n* tte task, (c) If you find yourself in the enviable position of 

By this stage, you're well steeped in evaluation lore and it s time to 
act We have a few suggestions, however, for the armchair evaluator: 

Alkin M C (1990). Debates on evaluation. Newbury Park, CA: Sage 
For those who haven't outgrown the 

this book has interviews with prominent evaluators throughout 
the nation on such topics as evaluation theory, use and ethics. 
As what these evaluators think eventually gets implemented in 
^^mS^line- for schools, we have an interesting peek at 
the future as well as at current dilemmas. 

Bank A., & Williams, R. C. (Eds.). (1985). Information systems and 

school improvement. New York: Teachers' College Press 

Computomania peaked about this time, so we ^ve a s Ul-curr^t 
view of how to use computers for school improvement. Sirotnik and 
Stein7chapter, "Making Sense Out of Comprehensive School 
Based information Systems:" provides an in-depth view of how one 
high school consolidated information needs. You'll also find the 
chapters by Bill Cooley and Robert Blum on automating the ele- 
mentary school and creating a school profile germane. 
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Resource A: Sample Parent Survey 



Name: 

Phone: ( ) 
Student's Name: 



F[ ] M[ ] Date: 



. Interviewer: 



Relationship to student (mother, father, guardian, etc.): 
. F[ ] M[ ] School/Teacher: ± 



Student's Grade Level: Language of Interview: Spanish English Other: 

1. What do you know about the Model Technology Program at your school? (Write exact quote) 

2. Please answer the following questions by indicating the number that corresponds to 
your response. 





Almost 
Never 


Rarely 


Some- 
times 


Often 


Almost 
Always 


Not Sure/ 
Don't 
Know 


a. My child enjoys going to school. 


1 


2 


3 


4 


5 


6 


b. My child talks to me about what 
s/he does in school. 


1 


2 


3 


4 


5 


6 


c. My child likes to do schoolwork 
on computers. 


1 


2 


3 


4 


5 


6 


d. My child likes to do schoolwork 
with video equipment. 


1 


2 


3 


4 


5 


6 


e. My child talks to me about his/her 
schoolwork with computers and 
other technology. 


1 


2 


3 


4 


5 


6 


f. I am satisfied with how much my 
child learns in school. 


1 


2 


3 


4 


5 


6 



3. In which three academic subjects would you like to see your child improve the most? PROBE 
a. English language skills e . Science i. other 

f. Spelling 

g. Reading 

h. Speaking 



a. 
b. 
c. 
d. 



Spanish language skills 
Math 

. Social Studies 



k. 



. other , 
. other . 



4. How satisfied are you with your child's progress in the following areas: 





Not at 
All 


Very 
Little 


Some 


A Lot 


A Great 
Deal 


NA/ 
Don't 
Know 


Reading 


1 


2 


3 


4 


5 


6 


Math 


1 


2 


3 


4 


5 


6 


Writing 


1 


2 


3 


4 


5 


6 



Copyright© 1992 by The Regents of the University of California 
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Science 


1 2 


Q 

0 




fi 6 


Social Studies 


1 £• 


Q 
0 




5 6 


Peer Relationships 


1 2 


3 


4 


5 6 


Self-Confidence 


1 2 


3 


4 


5 6 


Cultural Sensitivity 


1 2 


3 


4 


5 6 


Motivation for Learning 


1 2 


3 


4 


5 6 



The following questions are intended to gather information about parents as a group, 
member, answers to the next questions as well as to all of the questions you have an- 
swered will be kept strictly confidential. 

5. How long have you lived in this community? years 

6 What language(s) is (are) used primarily in your home? 

[ ] [ 1 M [ 1 

Most l y Mostly About the same Other language(s) 

English Spanish amounts of English Please Specify: 

and Spanish 



7. What is the highest grade level in school that you have completed? 



Where did you go to school? 



8. What is your marital status? 
[ ] Married [ ] Divorced 



[ ] Widowed [ ] Separated [ ] Single 



9. What is the employment status of the adults in your household? 

(Count adult #1 as respondent) 
Adult #1: Employed P-T [ ] F-T [ 
Adult #2: Employed P-T [ ] F-Tf 
Adult #3: Employed P-T [ ] F-T [ 
Adult #4: Employed P-T [ ] F-T[ 



Unemployed [ 
Unemployed [ 
Unemployed [ 
Unemployed [ 



In school P-T [ ] F-T[ ] 

In school P-T [ ] F-T[ ] 

In school P-T [ ] F-T[ ] 

In school P-T [ ] F-T[ 1 



10. Are there any additional comments or suggestions that you would like to make at this 
time? 



THANK YOU! 



Copyright © 1992 by The Regents of the University of California 

lit; 



Resource B: Sample Teacher Survey 

Please indicate your responses by checking, circling, or filling in the blanks. 



1. Sex: [ ]M [ ]F 

2. Ethnicity [ ] Caucasian [ ] Latino [ ] Native American 

[ ] Black [ ] Asian [ ] Other 

3. How many years including this one have you participated in the project? 



Academic/Professional Background 



4. What is the highest degree you have received? 

[ ] Bachelor's + teaching credential [ ] Master's + units beyond 

[ ] Bachelor's + units beyond credential [ ] Doctorate 

[ ] Master's [ ] Other (specify) 

5. Please indicate which teaching credentials you have and specify the content area of 
specialization. (Check ALL that apply.) 

[ ] General Elementary [ ] Single Subjects 

[ ] General Secondary [ ] Bilingual 

[ ] Special Emergency [ ] Other 

[ 3 Multiple Subject 

6. How many years of teaching experience do you have? years 

7. How many years have you taught bilinguaVLEP/bicultural students (including this 
year)? years 



8. Please characterize your current class(es): 

a. Grade level: 

b. Language(s) of instruction: 1. Mostly Spanish 2. Mixed 3. Mostly English 

c. Subject: (Junior High only) 

9. How would you characterize your effectiveness in teaching language arts to your 
current students? 



Low 
1 



2 



3 



4 



Extremely 
High 
5 



125 
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LZ ° TRACKING YOUR SCHOOL'S SUCCESS 

Project Effectiveness 

10. Please rate the effectiveness of the project in each of the following areas: 



Poor 




Adequate 




Very 
Effective 


NA 


a) Communication with team members 1 


2 


3 


4 


5 


6 


b) Communication across tpamq 1 


O 


Q 
O 


4 


5 


6 


c) Motivating teachers to usp 

technology in their classrooms 1 


2 


3 


4 


5 


6 


d) Making technology available in 

classrooms 1 


2 


3 


4 


5 


6 


e) Involving teachers in project 

decision making 1 


2 


3 


4 


5 


6 


0 Project management and 

organization 1 


2 


3 


4 


5 


6 


g) Project leadership 1 


2 


3 


4 


5 


6 


h) Project in-services 1 


2 


3 


4 


5 


6 


i) Work-study sessions 1 


2 


3 


4 


5 


6 


j) Operation of your Tech Center 1 


2 


3 


4 


5 


6 


k) Overall effectiveness of the project 1 


2 


3 


4 


5 


6 


Training 












11. How effectively do you feel you have been trained by the program in the following areas? 

Very 

Poor Adequate effective NA 


a) Curriculum development 1 


2 


3 


4 


5 


8 


b) Use of equipment 1 


2 


3 


4 


5 


6 


c) Use of software 1 


2 


3 


4 


5 


6 



d) Language arts instruction 12 3 



e) Integrating technology and 

language arts 12 3 

f) Using technology with language 
minority students to meet your 

__bilin gual program's g oals 12 3 



4 5 6 



4 5 6 
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12. How much follow-up and assistance has the project provided you in the following areas? 

A Great 

None Some Deal NA 



a) Use of technology in instruction 


1 


2 


3 


4 


5 


6 


b) Planning and developing curriculum 
units and activities 


1 


2 


3 


4 


5 


6 


c) Becoming familiar with and 
learning about technology 


1 


2 


3 


4 


5 


6 


d) Training students in equipment 
usage 


1 


2 


3 


4 


5 


6 


e) Integrating the use of technology 
with bilingual instruction 


1 


2 


3 


4 


5 


6 


Use of Technology 



13. Approximately how much has your class used the Tech Center this school year? 
hours/week 



14. On average, how much do you use technology as part of your regular classroom 
program (not including Tech Center time)? Please indicate which of these are 
permanently in your classroom: 

PERMANENTLY IN 
MY CLASSROOM 



Video equipment hours/week [ ] 

Computer equipment hours/week [ ] 

Laser disc equipment hours/week [ ] 

Other: __ _hours/week [ ] 



15. a. On average, how many hours a month do you spend doing the following project- 

related activities? 

hours a) Participating in project meetings and in-services 

hours b) Planning/developing instructional activities/curriculum units 

hours c) Becoming familiar with technology on your own 

hours d) Helping students in the use of the equipment during nonclass time 

hours e) Other (t ^scify) 

b. About what percentage of this time is unreimbursed, nonschool time? % 

16. How many times this school year have you shared your products and ideas with others? 

a. At conferences 

b. With non-MTS colleagues_ 

c. Informal presentations 
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TRACKING YOUR SCHOOL'S SUCCESS 



Effects of Project 

17. To what extent has your involvement in the project affected your expectations for 
your students' achievement? 

Greatly Slightly Has Not Slightly GreaLy 

Lowered Lowered Affected Increased Increased 

1 2 3 4 5 

Please explnin: — — 



18. How much of a difference (if any) has participating in the project had on your 
students' school-related attitudes and behaviors? 

A Great 

None Some Deal 



a) Increased lik>*ig of school 


1 


2 


3 


4 


5 


b) Improved language arts skills 


1 


2 


3 


4 


5 


c) Improved confidence as a learner 


1 


2 


3 


4 


5 


d) Increased interest in technology 


1 


2 


3 


4 


5 


e) Improved student-teacher rapport 


1 


2 


3 


4 


5 


0 Increased participation in class 


1 


2 


3 


4 


5 


g) Improved student-student cooperation 




1 


2 


3 


45 



h) Other (specify below): 1 



19. Please rate the extent to which your involvement with the project has affected the following: 

A Great 

None Some Deal 



a) Improved my overall teaching effectiveness 1 


2 


3 


4 


5 


b) Improved my language arts instruction 1 


2 


3 


4 


5 


c) Changed my perceptions about my students' 
learning abilities 1 


2 


3 


4 


5 


d) Changed my instructional methods 1 


2 


3 


4 


5 



Please explain these ratings: 



a) 
b) 
c) 
d) 
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20. Please rate the extent to which you agree or disagree with each of the following 



129 



statements: 

Strongly 

Disagree Disagree Neutral Agree 


Strongly 
Agree 


_\ t ow% nnfVtnQiaQtir nhnnt narticiDation 
in this project. 


3 4 


0 


U\ A/lminiafrafnrQ fit TY1V SpHoOl QX6 

eenerallv supportive of the project. 1 2 


3 4 


K 
U 


c) My principal provides strong support 

for my teaching. 1 


3 4 


K 
O 


d) I feel a lot of stress at work. 1 2 


3 4 


0 


e) I am satisfied with my work environment. 1 2 


3 4 


5 


0 I feel excited by my students' 

accomplishments. — 


3 4 


5 


g) The project is helping me develop 

professionally. 1 


3 4 


5 


m t am /Wplnniner teacher training skills. 1 2 


3 4 


5 


21 How satisfied are you with your students' progress in the following areas? 

Highly 

Satisfied Neutral Dissatisfied 


Reading 5 4 3 


2 1 




Math 5 4 3 


2 1 




Writing 5 4 3 


2 1 




Science 5 4 3 


2 1 




Social Studies 5 4 3 


2 1 




Peer Relationships 5 4 3 


2 1 




Self-Confidence 5 4 3 


2 1 




Cultural Sensitivity 5 4 3 


2 1 




Motivation for Learning 5 4 3 


1 2 1 





22. Please share your impressions of the project's accomplishments and/or how it may be 
improved: 



THANK YOU!! 
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Resource C: Sample Student Survey 



The survey you are about to complete will ask you questions about you and about your 
school. This is NOT a test. There are no right or wrong answers. The survey will give you an 
opportunity to express how you feel about what happens in your classes and around school 
1 hat is why it is important to answer the questions as truthfully and as carefully as possible." 

Do Not Write on These Pages 

MARK YOUR ANSWERS ON THE ANSWER SHEET PROVIDED. You will notice that 
answers go from 1 to 5 or from 6 to 10. This does not matter. Simply choose the one answer 

Swft ? foS f^KS™^" 68 * 1011 - MARK ° NLY 0NE LETTER 0N THE ANSWER 
SHEET FOR EACH QUESTION. For example, if you chose answer 5 for question number 5 
you would mark the answer sheet like this: 

1 2 3 4 5 
O O O O • 

Or, if you chose answer 9 for question number 6, you would mark the answer sheet like this: 

. 6 7 8 9 1 0 
b O O O • O 

Remember, mark only one letter on the answer sheet for each question. If there are any words 
or questions you don't understand, please raise your hand and ask for help. 
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1. Sex: 

1. Male 

2. Female 

2. In addition to English, what other languages are spoken in your home? 

1. None 

2. Spanibh 

3. Vietnamese 

4. Chinese 

5. Other 

3. Living situation: 

1. With two parents (includes stepparents) 

2. With one parent only (mother or father only) 

3. Guardian(s)/foster parents 

4. Alone or with friends 

5. Other 

4 ' school h yl™ Y h ° UrS " d ° y ° U USUaHy Spend WOrkh * on a J° b ^ring the 

1. None: I am not employed during the school year 

2. About 10 hours or less. 

3. About 15-20 hours 

4. About 20-30 hours 

5. More than 30 hours 

5. How many hours do you watch television each day? 

1. None 

2. About 1 hour 

3. About 2-3 hours 

4. About 4-5 hours 

5. More than 5 hours 

Choose the ONE answer that best completes each of the following sentences. 

6. If I could do anything I want, I would like to 

1. quit school as soon as possible. 

2. finish high school 

3. go to trade/technical school or junior college 

4. go to a 4-year college or university 

5. don't know 
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TRACKING YOUR SCHOOL'S SUCCESS 



7. I think my parents would like me to 

1. quit school as soon as possible 

2. finish high school 

3. go to trade/technical school or junior college 

4. go to a 4-year college or university 

5. don't know 

8. Actually, I will probably 

1. quit school as soon as possible 

2. finish high school 

3. go to trade/technical school or junior college 

4. go to a 4-year college or university 

5. don't know 

9. How comfortable do you feel about choosing a future career goal at this point in your 
life? 

1. Very uncomfortable 

2. Uncomfortable 

3. Neither uncomfortable nor comfortable 

4. Comfortable 

5. Very comfortable 

The following sentences describe some of the ways in which people might think about 
2li3S each of the following sentences carefully and mark the number on the 
answer sheet that tells how much it is like you. 

Look at the following practice sentence and mark the letter on the answer sheet that tells 
how much you agree or disagree with the sentence. 

PRACTICE strongly MUdly Not MUdly Strongly 

Agree Agree Sure Disagree Disagree 

« , i 5 4 3 2 1 

I am good at art ° 

Tf vou choose "Strongly Agree," you're saying that you are very good at art. If you choose 
3^ -aySg that you are OK at art. If you choose 'Mildly Jisagree," you're 
saying thaTyou are not too good at art. If you choose "Strongly disagree," you're saying that 
you are very poor at art. 

Strongly Mildly Not Mildly Strongly 
Agree Agree Sure Disagree Disagree 

10. I'm popular with kids my own age. 5 4 3 2 1 

11. Kids usually follow my ideas. 5 4 

12. Most people are better l«ked 4 3 2 1 

than I am. 5 1 q 2 1 

13. It is hard for me to make friends. 5 4 6 
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14. I have no real friends. 

15. I'm not doing as well as I'd like to 
in school. 

16. I am a good reader. 

17. I'm proud of my school work. 

18. Fm good at math. 
I'm doing the best work that I can. 
I am able to do schoolwork at least 
as well as other students. 
My grades are not good enough. 
I'm always making mistakes in my 
schoolwork. 
I am a good writer. 
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19. 
20. 

21. 
22. 

23. 



5 
5 
5 
5 



5 
5 

5 
5 



4 
4 
4 
4 
4 

4 
4 

4 

4 



3 
3 
3 
3 
3 

3 
3 

3 
3 



2 
ii 

r 

2 
2 
2 

2 
2 

2 
2 



QUESTIONS ABOUT YOUR SCHOOL LIFE 

How much do the following words describe most of the teachers at this school? 



24. Friendly 

25. Helpful 

26. Have high hopes for us 

27. Talk to us 

28. Let us talk to them 

29. Care about us 

30. Do a good job 



Very 
Much 
5 
5 
5 
5 
5 
5 
5 



Pretty 
Much 
4 
4 
4 
4 
4 
4 
4 



Some- 
what 
3 
3 
3 
3 
3 
3 
3 



Only a 
Little Bit 
2 
2 
2 
2 
2 
2 
2 



Not 
at All 

1 

1 

1 

1 

1 

1 

1 



How much do the following words describe how you feel about most of the students at this 
school? 

Very 
Much 

31. Friendly 5 



32. Helpful 

33. Have high hopes 

34. Smart 

35. Talk to each other 

36. Care about each other 

37. Competitive 



5 
5 
5 
5 
5 
5 



Pretty 
Much 
4 
4 
4 
4 
4 
4 
4 



Some- 
what 
3 
3 
3 
3 
3 
3 
3 



Only a 
Little Bit 
2 
2 
2 
2 
2 
2 
2 



Not 
at All 
1 
1 
1 
1 
1 
1 
1 



38. The most popular students in this school are (choose only one answer): 



1. Athletes 

2. Smart students 

8. Members of student government 

4. Good-looking students 

5. Wealthy students 
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TRACKING YOUR SCHOOL'S SUCCESS 



Yes 



No 
2 
2 
2 
2 
2 



39. I participate in sports teams/drill team/flags/cheerleading 

40. I participate in student government 

41. I participate in music, band, drama, or other arts 

42. I participate in honor society 

43. I participate in school clubs/community service activities 

XZ^vbSJ* may be problems at this sch001 - How much d ° y™ think -* 

Not a Minor Major 

44. Student misbehavior (fighting, stealing, gangs, truancy, etc.) T * ^T* ***** 

45. Poor courses or not enough different subjects offered i o 

46. Prejudice/racial conflict 

47. Drugs 

48. Alcohol 

49. Poor teachers or teaching 

50. School too large/classes overcrowded 

51. Teachers don't discipline students 

52. Poor or not enough buildings, equipment, or materials 
5d. ine principal and other people in the office who run the school 

54. Poor student attitudes (poor school spirit, don't want to learn) 

55. Too many rules and regulations 

56. How the school is organized (class schedules, not enoudi 
time for lunch, passing periods, etc.) 1 2 3 

ISSUES AND PROBLEMS 

Read each one of the following sentences carefully and choose thp lptrm- tkot * Q iio u 

Strongly Mildly Not 

t*u- i . j Agree Agree Sure 

1 think students of different races 

or colors should go to school together. 543 
There are places at this school 
where I don't go because I am 

afraid of other students. 543 
I do not have enough time to do 

my schoolwork. 5 4 3 

Some of the things teachers want 

me to learn are just too hard. 5 4 3 

61. If I had my choice, 1 would go to 

a different school. 5 a 0 



3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 



57. 



58 



59 



60. 



Mildly Strongly 
Disagree Disagree 
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62. There are things I want to learn 
about that this school doesn't teach. 5 

63. It's not safe to walk to and from 
school alone. & 

64. I have trouble reading the books 

and materials in my classes. 5 

65. The grades or marks I get help me 

to learn better. 5 

66. I like school. 5 

67. The grades or marks I get in class 
have nothing to do with what I 

really know. 5 

68. I have to learn things without 

really knowing why. 5 

69. Parents should have a say in 

what is taught at school. 5 

70. It is easy for me to get help from 
a counselor when planning my 

school program. 5 

71. We are not given enough freedom 

in choosing our classes. 5 

72. We are not given enough freedom 

in choosing our teachers. 5 

73. If I have a personal problem, it 
would be easy for me to get help 

from a counselor. 5 

74. If you don't want to go to college, 
this school doesn't think you're 

very important. 5 

75. Students should have a say in 

what is taught at this school. 5 

76. A person is foolish to keep going to 
school if he/she can get a job. 5 

77. If I need help planning for a career, 
it would be easy for me to get help 
from a counselor. 5 

78. I like the way the school looks. 5 

79. It is easy to get books from the 
school library. 5 

80. Things in the school library are 
useful to me. 5 

81. Materials in the Career Guidance 
Center (CGC) are useful to me. 5 
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QUESTIONS ABOUT TEACHING, LEARNING, AND CLASSROOM WORK 

All schools teach pretty much the same things, but they may think some things are more 
important than others. 

82. Which ONE of these does this school think is the most important thing for students'? 
(Choose only one.) 

1. To work well with other people. 

2. To learn the basic skills in reading, writing, arithmetic, and other subjects. 

0. 1 o become a better person. 
4. To get a good job. 

83. If you had to choose only the ONE most important thing for you, which would it be? 
(Choose only one.) 

1. To work well with other people. 

2. To learn the basic skills in reading, writing, arithmetic, and other subjects, 
o. To become a better person. 

4. To get a good job. 

In general, how do you like the following subjects? 

Like Very Like Unde- Dislike Dislike Very 

aA „ .. . Much Somewhat cided Somewhat Much 

84. English 5 4 

85. Mathematics 5 4 

86. Social Studies (history, geography, 
government, etc.) 5 4 

87. Science 5 4 ^ ^ 

88. Computer Education 5 4 3 2 

89. The Arts (art, crafts, music, drama, 
dance, creative writing, filmmaking, 
photography) 5 

90. Foreign Language 5 4 

91. Vocational/Career Education 
(shop, business education, home 
economics, etc.) 5 4 

92. Physical education 5 4 

93. How many hours of homework do you do each day? 

1. None 

2. About 1 hour 

3. About 2-3 hours 

4. About 4-5 hours 

5. More than 5 hours 



2 
2 

2 
2 



4 3 2 

3 2 
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94. In general how often do you do your homework? 



1. All of the time 

2. Most of the time 

3. Sometimes 

4. Seldom 

5. Never 

Sf;. How soon do teachers usually return your work? 

1. The next day 

2. 2 days later 

3. 3 days later 

4. 4 days later 

5. 5 days later or more 

96. When you make mistakes in your work, how often do teachers tell you how to do it 
corretly? 

1. All of the time 

2. Most of the time 

3. Only sometimes 

4. Seldom 

5. Never 

97. How often do your parents or other family members help you with your schoolwork? 

1. All of the time 

2. Most of the time 

3. Only sometimes 

4. Seldom 

5. Never 

Listed below are four ways students can work in a classroom. Choose the number on the 
answer sheet that tells how much you like or would like to work in each way, even if you 

don't do so now. ,„ TT 

Like Very Like Unde- Dislike Dislike Very 

Much Somewhat tided Somewhat Much 

98. Alone by myself 5 4 3 2 1 

99. With the whole class 5 4 3 2 1 

100. With a small group of students 

who know as much as I do 5 4 3 2 1 

101. With a small group of students, 
some who know less, some who 
know as much, and some who 

know more than I do 5 4 3 2 1 
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TRACKING YOUR SCHOOL'S SUCCESS 



Listed below are some things that might be used in a class. Choose the number on the answer 
sheet that tells how much you like or would like to use each thing, even if you don't use it in 
a classroom. 

Like Very Like 
Much Somewhat 



102. Textbooks 

103. Other books 

104. Worksheets 

105. Films, filmstrips, or slides 

106. Games or simulations 

107. Newspapers or magazines 

108. Tape recordings or records 

109. Television/video 

110. Calculators 

111. Globes, maps, and charts 

112. Animals and plants 

113. Lab equipment and material 

114. Computers 



5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 



Unde- Dislike Dislike Very 
tided Somewhat Much 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 

4 3 2 



Listed below are some things that you might do in a class. Choose the number on the answer 
sheet that tells how much you like or would like to do each thing, even if you don't do it in 
class. 

Like Very Like 
Much Somewhat 



115. Listen to the teacher 5 

116. Go on field trips 5 

117. Do research and write reports, 
stories, or poems 5 

±18. Listen to student reports 5 

119. Listen to speakers who come to 
class 5 

120. Have class discussions 5 

121. Build or draw things 5 

122. Do problems or write answers 

to questions 5 

123. Take tests or quizzes 5 

124. Make films or recordings 5 

125. Act things out 5 

126. Read for fun or interest 5 

127. Read for information 5 

128. Interview people 5 

129. Do projects or experiments that 

are already planned 5 

130. Do projects or experiments that 

I plan 5 



4 
4 

4 
4 

4 
4 



4 
4 
4 
4 
4 
4 



Unde- 
cided 
3 



Dislike Dislike Very 
Somewhat Much 
2 
2 



3 
3 

3 
3 



3 
3 
3 
3 
3 
3 



2 
2 

2 
2 
2 

2 
2 
2 
2 
2 
2 
2 
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Please indicate how important each of the following items was in your choice of classes here 
at Royal High School. 

Very 

Important Important 



Not 
Sure 
3 



5 
5 



4 

4 
4 



131. Taking classes from teachers I like 

132. Being in the same classes as 
my friends 

133. Completing graduation 
requirements 

134. Learning skills for a future job 

135. Taking classes that will help me 
be a better person 

136. Being challenged by taking hard 
subjects 

137. Taking classes that will prepare 
me for the future 

138. Getting a wide variety of classes 

139. Preparing for college 

140. Taking classes that require 
little work 

141. Avoiding subjects I don't like 

142. Taking classes that are popular 5 4 3 

143. Taking classes my parent(s) 

.onsider important 5 4 3 

144. Taking classes where I can get 

good grades 5 4 3 

QUESTIONS ABOUT THE LEARNING RESOURCE CENTER (LRC) 

145. Have you heard of the Learning Resource Center? 



Not Very Un- 
important important 
2 



5 
5 



5 

5 



4 
4 



4 
4 



3 
3 

3 



3 
3 



3 
3 



2 
2 



2 
2 



2 
2 



1. 
2. 



Yes 
No 



146. If yes, how often have you gone with your classes to the Learning Resource Center? 

1. Never 

2. Only once or twice 

0. About once or twice an month 

4. About once or twice a week 

5. Almost every day 

147. How often have you gone to the Learning Resource Center by yourself? 

1. Never 

2. Only once or twice 

3. About once or twice an month 

4. About once or twice a we( 

5. Almost every day 
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TRACKING YOUR SCHOOL'S SUCCESS 



If you have ever used the Learning Resource Center, have you used any of these services? 
(Answer yes or no for each of the following.) 

Yes 

Diagnostic testing for reading and math problems 
Entry testing for proper class placement 
Assistance with assignments from classroom teacher 
Work on tasks assigned by the Learning Resource Center 
After-school seminars 
Study hall 
SAT preparation 
Proficiency test preparation 
Use the computer 
Study skills 
Language laboratory 

Assistance in researching or typing papers 
Use the typewriter 
Receive individual tutoring 
Develop library/research skills 
Develop reading skills 
Develop writing skills 
Develop math skills 
Develop listening skills 
Develop test taking skills 

Have you received credit for Writing I through the 
Learning Resource Center? 

Have you received credit for Developmental Reading 
through the Learning Resource Center? 



148. 
149. 
150. 
151. 
152. 
153. 
154. 
155. 
156. 
157. 
158. 
159. 
160. 
161. 
162. 
163. 
164. 
165. 
166. 
167. 
168. 

169. 



No 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



Strongly Mildly 



170. The Learning Resource Center 
is helping students at Royal. 

171. Most students know about the 
resources available in the 
Learning Resource Center. 

172. I have been helped by the services 
of the Learning Resource Center. 

173. I am comfortable about using the 
services of the Learning Resource 
Center. 

174. My work in the Learning 
Resource Center has helped me 
in my courses. 

175. My work in the Learning 
Resource Center has made me 
feel more secure about my ability 
to do the work assigned by 

my teachers. 



Agree Agree 



Not Mildly Strongly 
Sure Disagree Disagree 



3 
3 



2 ' 
2 



1 
1 
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QUESTIONS ABOUT THE CAREER MAGNET SCHOOL 

Strongly Mildly Not Mildly Strongly 
Agree Agree Sure Disagree Disagree 

176. I understand what the Career 
Magnet School program is 

trying to do. 5 4 3 2 1 

177. I would like more information 

about the Career Magnet Schools. 5 4 3 2 1 
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Worksheet 1.1 

Evaluation Focus and Priority Questions 




Priority Rating 


Questions 


Low Medium 


High 


Required 


Outcomes: 










Curriculum: 










Instruction: 










Staff: 










Parents and Community: 


i 




i 
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Worksheet 2.1 

Tracking Strategies 


Evaluation 
Questions 


1. 

2. 
3. 
4. 
5. 
6. 


Questions 


Evidence of Progress 


Source of Information 
(Instruments) 


Standards of Success 
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Worksheet 3.1 

Instrument Blueprints 



Instrument 



Questions Asked 



Portfolios 



Parent Questionnaire 



Student Questionnaire 



Faculty Interviews 



Standardized Nationally Normed 



State Assessment 



Competency 



Other 



ERLC 



16 



o 

o 

3. 
oq 

0 

M 

CO 
CD 

to 

H 
W 

(D 
OQ 
<T> 

c* 

CO 

o 



o 
C 

t-i. 

o 

O 

5 



Worksheet 3.2 

Data Collection Management Plan 



Task by Instrument 



Instrument: Math Portfolios 

Select group to develop/choose 

Develop task and criteria 

Produce 

Distribute 

Administer 

Score 

Analyze and interpret 



Instrument: Math Chapter Tests 

Select group to develop/choose 

Develop tasks and criteria 

Produce 

Distribute 

Administer 

Score 

Analyze and interpret 



Instrument: State Assessment 

Select group to develop/choose 

Develop tr-ks and criteria 

Prod oce 

Distribute 

Administer 

Score 

Analyze and interpret 



Person Responsible 



Instrument: ITBS 

Select group to develop/choose 

Develop tasks and criteria 

Produce 

Distribute 

Administer 

Score 

Analyze and interpret 



Sampling 



Sept. 



Oct 



Nov. 



Dec. 



Jan. 



Feb. 



Mar. 



Apr. 



May 
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Worksheet 4,1 

Summarizing Data 


Instrument 


Score Selected 


Siimnifirv Mpthnri 


Questionnaire/Survey 






Nationally Normed Test 






Selected Response Curriculum- 
Related Tests 







Performance Tests 



Interviews (open-ended responses) 



Archival Information 
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Worksheet 5.1 

Data Analysis for Specific Evaluation Questions 



Question: In what areas are we in need of improvement? 

Instrument Score Used Analysis 



Question: How well are we implementing a process-based approach to writing? 

Instrument Score Used Analysis 



Question: Is student writing improving? 

Instrument Score Used Analysis 



Question: Do students feel positive about the writing program? 

Instrument Score Used Analysis 
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Worksheet 6.1 

Communicate and Act on Findings 






Report Format 




Audiences 


Key Concern(s) 


Formal 


Informal 


Date 


Teachers: 










Students: 








— , 


Parents: 










School Board: 










Community: 










District/Other 
Administrators: 










Other: 




i 






Key for entering report format: 

H a Handouts: tables, sample items/performance tasks, score summaries, and so on 
N = Nev/sletter: school, district, PTA 

G = Oral, informal report: faculty meetings, district meetings 

P = Presentation: formal, with invited audience, agenda, overheads, and so on 

T = Technical report: formal, written report following format of research report 
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Report to Parents 

Brighton School 

727 S. Brighton Ave, 
Covina, CA 91723 



Mrs. Georgia Florentine, Principal 



School Profile 



Brighton School is one of twelve K-6 schools in the 
Covina-Valley Unified School District. The enroll- 
ment as of October 1 , 1991 was 629 students. The 
ethnic makeup of the school is indicated on the 
following chart: 




Ethflic Distribution 



U Whiu 

■ Wwk 

EJ Ailin 

G3 nupino 



28 7* 
5 9% 
fl v% 

10% 



Note : Because of rounding, the percentages may 
not total to exactly 100%. 

Brighton School was built in 1952. Since 1987 , 5 
new relocatables have increased the number of 
classrooms to 23. These additions allow us to 
accommodate the growth in student population in 
our community. 

A California Compact with the South Hills Kiwanis 
Club was recently formed to set up a buddy tutoring 
program. In addition, enrichment for Brighton stu- 
dents is provided by a partnership with the local 
Carl's Jr. 



Instructional and Support Staff 

Brighton's teaching staff includes 22 regular educa- 



November 1991 

tion teachers, 1 special education teacher, instru- 
mental/choral/classroom music teachers and 1 Title 
V!l resource teacher. All teachers have assignments 
within their credential authorization. 

Additional personnel include the school principal 
secretary, office clerk, custodian, janitor, food ser- 
vices staff, and 17 instructional aides and a home 
school assistant. 

In audition, psychologists, nurses, language and 
speech specialists, an audiologist, and an adapted 
P.E. specialist are available to provide support in 
meetingthe academic, personal, and emotional needs 
of our students Child abuse prevention programs 
and a crisis re^nse team are also in place. 

Advanced degrees are held by 60% of the certifi- 
cated stalf. 



Quality of Instruction and Leader- 
ship 

All students are expected to develop effective oral 
and written communication skills, solve mathemati- 
cal problems and think logically. Teachers use in- 
structional strategies that promote active involve- 
ment of all students and administrators serve as 
instructional leaders. A continuous effort is made to 
ensure that all students regardless of special need, 
e.g., limited English speaking, special education, 
gifted, have access to the core curriculum. 

The quality of the school's instructional program is 
assessed every 4 years using the State Consoli- 
dated Program Review . During the last review, the 
visiting committee stated that the strengths of the 
school were: 

Enthusiastic, positive learning environ- 
ment. 

* Strong commitment to academic excel- 
lence 
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Academic Performance 

Because of statewide cutbacks within California 
Assessment Program during the 1990-91 school 
year, the CAP exams were not administered. The 
following information is based on the latest CAP 
scores. Third and sixth graders at Brighton School 
tookthe California Assessment Program(CAF\ .>xam 
in May of 1990. The exam uses ^ scale from 1 00 to 
400, with the statewide average set at 250 the first 
year the test was administered. In May 1990, third 
graders scored 310 in reading, 307 in written lan- 
guage, and 320 in mathematics. The sixth graders 
scored 328 in reading, 299 in written language, and 
330 in mathematics. The following graphs show 
three-year comparisons of scores: 




Wt It If 19 



□ 1987-86 S 1988-80 



IQS9-Q0 



ELEMENTARY GR. 6 CAP SCORES 



400 T 



319 330 

:m ?99 jocl 




Student Attendance 

During the 1990-91 school year, an average of 
94.0% of all students at Brighton School were in 
attendance. It is estimated that 1 % of student atten- 



dance days were generated by students who did not 
attend full minimum instructional days because of 
unexcused absences. Parents are encouraged to 
have their sons and daughters attend every day 
during the school year. Success in school depends 
on the ability to acquire specific skills and knowledge 
in all classes every day of the year. 



Class Size 

California ranks as one of the highest in the nation 
in the number of students per teacher . As of 
October 1, 1991 Brighton's average class size was 
28.6 students per teacher. The average class size 
for K-3 was 27.8 and 4-tt was 29.6 students per 
teacher. Class enrollment is balanced to keep class 
size as low as possible. 



Substitute Teachers 

When teachers are absent, it is the District's goal to 
employ the highest quality substitutes available. 
Because there is a general shortage of substitute 
teachers throughout California, the District main- 
tains a competitive substitute salary rate and has an 
ongoing recruitment program to employ qualified 
substitute teachers. All substitute teachers hold a 
valid California teaching credential based upon a 
bachelor's degree and have passed the California 
Basic Educational Skills Test (CBEST) when that is 
a requirement; many of them, including those who 
have retired from Covina-Valley, have regular teach- 
ing experience. 



District Expenditure 

The C-VUSD is spending an average of $4,107 per 
student for all educational services including sala- 
ries, instructional materials, maintenance, transpor- 
tation, and capital expenses. 

In addition, our school receives specialized funds for 
the following programs; 

Chapter 1 

Chapter 2 

Gifted and Talented Education (GATE) 
Limited English Proficient (LEP) 
School Improvement (SI) 
Title VII 

Special Education 
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• High teacher involvement in staff devel- 
opment. 

• Leadership and support from many on- 
site Mentor teachers. 

• Special needs services by GATE, LEP, 
RSP, Speech and Chapter 1. 

Climate for Learning 

Brighton's goal is to provide a positive climate for 
learning in order to assure student achievement, 
enhance self-esteem, and develop positive social 
behavior. To reach this goal, Brighton School has 
the following opportunities for student participation 
and recognition: 

• Principal's Awards 

• Monthly Citizenship Assemblies 

• Performing Arts 

• Perfect Attendance Awards 

• Presidential Academic Fitness Awards 

Students are expected to maintain appropriate be- 
havior at all times because classroom disruptions 
interfere with learning. School rules and procedures 
are distributed and discussed on a regular basis. 
Parent support in recognizing their children for posi- 
tive efforts in this area benefits the educational 
program. 



Textbooks/Instructional Materials 

Covina-Valley USD follows a cycle tor adoption of 
State approved textbooks. Each major curricular 
area is reviewed every seven years by representa- 
tive teacher committees and modified as necessary. 
Appropriate textbooks and other instructional mate- 
rials are purchased to support the program. 

A math program was implemented with the purchase 
of a new textbook series in 1988. Core literature 
books were purchased and are now being used in the 
classroom. In 1990-91, a new language arts pro- 
gram stressing quality literature and frequent writing 
was implemented and new material was purchased. 
This year a new social science program has been 
implemented and new materials purchased. 

The library contains 6,500 volumes and is staffed by 
an instructional aide. There are 32 computers, dis- 
tributed throughout the school in individual class- 
rooms. The resource room contains a wide variety 



of supplemental materials. A valuable collection of 
classroom sets of literature books is also available 
forcheckout. LEP and Chapter 1 students utilize this 
facility daily. 

Curriculum Improvement and 
Training 

Each major subject area is reviewed and revised 
every seven years. Alter curriculum is revised and 
new courses of study or course descriptions are 
written, appropriate instructional materials are se- 
lected and staff in-service programs designed. New 
programs are developed based on the latest re- 
search in curriculum and instruction and State frame- 
works. 

Many in-service opportunities are available for 
teachers and administrators during school, after 
school and during the summer. Teachers, admin- 
istrators, and support staff receive training at the 
school, the District Office or attend workshops or 
conferences throughoul Southern California. Pre- 
senters are often Covina-Valley Unified School District 
teachers, many working as Mentor Teachers with 
special projects such as Math Manipulates, Co- 
operative Learning, The Writing Process, The Use of 
Literature in the Classroom and Learning Styles. 
Most in-service opportunities are directly related to 
new directions in the curricular program and new 
teaching strategies. 

New teachers receive training before their first teach- 
ing day and continual support from many sources 
including Mentor teachers. 



Teacher Evaluation and Profes- 
sional Improvement 

The purpose of the teacher evaluation process is to 
improve instruction. While all teachers are observed 
and evaluated informally on an ongoing basis as 
needed, under the law nontenured teachers are 
formally evaluated in writing every year and tenured 
teachers at least every other year. Examples of 
evaluation areas are instructional methodology, 
suitable learning environment, and classroom man- 
agement. Teachers and the principal receive appro- 
priate training in these areas. 
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State law require* that comparative talary and budget Information — which is taken from documents available to the general public — be displayed in each school's Report to Parents: 



SeJarvCj 

(196*90) 



CovtnaVaRev't 
Averag e 



Comparison — State Average 
Districts over 1.500 ADA ' 



Comparison — State Average 



Teachers 




Annual Salary 




Annual Salary 




Daly Pay 


Annual ^tanr Bum 




Beginning 
Mid-range 
Highest 


124.970 
M3.I50 
$47,440 


$136 
$235 
S258 


$23,779 
$96,952 
$45,284 


$24,273 
$37,967 
$47,352 


$131 
$205 
$256 


$20,266- $28,000 
$32,218- $43,241 
$41,046-$53.93g 


School Administrators (average) 




$64,027 


$291 


$57,284 


$59.e55 


$280 


$40.383-$68.670 


Superintendents 




$84,648 


$376 


$82,324 


$87,683 


$391 


$60,e95-$1 14,784 


BudoelPefcenUfM 

(1969-90) 
















For Teachers' Salaries 




41.37% 




44.01% 


44.25% 


38.79%-48.33% 


For Administrative Salaries 




4.46% 




5.69% 


5.82% 


4.73%-8.72% 



' ADA means Average Daily Attendance 



Safety 

As part of the school's commitment to a safe and orderly campus, Brighton School has an earthquake/disaster 
preparedness plan and conducts regular fire and emergency drills. 

Safety concerns, when identified by the District safety committee orthe school staff, are referred to the District 
Maintenance Department for correction. 



aivtNJtvjuity 

UNIFIED SCHOOL OMT' ICT 



BOARD OF EDUCATION 

Agnes F. Garwacki 
Mary L. Hanes, M.D. 
Richard K. Harder 
Gilbert R. Ramirez 
Ellen F. Sandt 

Jack H. Rankin, Ed.D. 
District Superintendent 
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Concerns Based Adoption Model 
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selection criteria, 118 

Cooley, B., 116 

Costs, 56, 61 

Credibility, 37, 39, 55 

Cultural bias, 34, 57 

Curriculum, 3, 4, 18, 22, 27, 29 



mathematics, 24, 87, 91, 97 
whole langauge, 23-24, 94 



Data 

analysis, 14, 79-98, 113, 115, 119 
collection, 6, 13-14 
disasters, 57, 61-62 
display, 82-85 
interpretation, 80, 113 
management, 47-63, 81 
scheduling, 55 
sources, 36 

summarization, 74-76, 81-82, 113 
Database, 115-116 
Decisions, 7, 9 
Dorr-Bremme, D., 8 



Education, current trends in, 25-26 
Eisner, E., 8 
Equity, 21 

Evaluation, 10-12, 13-15, 111 
automating, 114-117 
benefits, 120 

focus, 13, 17-32, 112, 115, 118 
purposes, 9, 16 
questions, 7, 28-30, 112 
steps in, 13-15 
timeliness, 5 
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worksheets, 142-149 
Evidence, 39. See also Data 
Excel, 117 



Feasibility (of measures), 38-39 
Formative evaluation, 20, 21-28, 45, 
94-97, 112 



Graue, M. E., 8 



Hardware, 116 
Hord, S. M., 24 



Instruction, 3, 18, 22, 25-27, 29 
Instruments, 39, 41, 72-73. See also 
Tracking Strategies; Testing 



Learning, 25 

Learning Research and Development 

Center (LRDC), 116 
Linn, R., 8 



Management (of evaluation), 47, 49, 

112, 115, 119 
Mathematics. See Curriculum 
McLaughlin, M. W., 24 
Model (of schooling), 19 
Monitoring (of program), 108, 114, 

115, 120 



Needs Assessment, 19-21, 45, 86, 

90-93 
Niemi, D., 73 



Observations, 39, 41, 43, 73 
Outcomes (student), 3, 18-19, 22, 
26-29 



Patton, M. Q., 9, 29 
Performance assessment, 38, 42, 57, 
66, 69 

Portfolios, 38, 43, 50, 67, 74 



Q and A, 117 

Questionnaires, 39, 41, 43, 48, 50, 
69, 123-129 



Reporting results, 14, 99, 100, 101- 

106, 114-115, 120 
Restructuring, 3, 17 



Sanders, N. M., 8 

School based management, 3, 17 

School climate, 3, 5, 18, 22, 27, 29 

Scoring, 14, 55, 65-76, 113, 115, 119 

Significance testing, 85-86 

Sirotnik, K. A., 8 

Software, 116-117 

Staff (as evaluation focus), 3, 5, 18, 

22, 27, 29, 
Stakeholders, 12 
Standards, 40-45, 88 
Statistical software, 115-116 



Templates, 115 

Testing, 8, 13, 42-43, 58-59, 70-73 
Timeliness, 52 

Tracking strategies, 11, 13, 33-46, 
89, 112. See also Testing 



Validity, 11, 37-38, 49 



Whole language. See Curriculum 

Williams, R., 116 

Word Perfect, 117 

Word processing, 115, 116 

Works (Microsoft), 117 

Writing samples, 73 



Parents, 3, 5, 18, 22, 27, 29, 106, 113 

Parent-Teacher Assocation (PTA), 99 Yamaguchi, E., 73 
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Tracking Your School's Success 
A Guide to Sensible Evaluation 

Joan L« Herman, Center for Research on Evaluation, UCLA 
Lynn Winters, The Galef Institute, Los Angeles, Calif. 

"Excellent Resource! HIGHLY recommended for those who are in the administrative arena 
and a MUST READ for those just entering or planning to enter the field of educational 
administration." Carl L, Lane, Principal 

Rancho-Starbuck Intermediate School, 
LowelJ.S.D., Whittier, Calif. 

"As a person beginning to understand the complexity of program evaluation, I think the 
authors have written a very helpful book. Their six-step model is a clear one with many 
strengths. I especially liked their introduction which answers the very obvious and crucial 

question(s): 'Why conduct an evaluation?' (and) 'Who should be involved?' In fact, 

their overview -Sensible Evaluation -is the key ingredient in program evaluation." 

Winston Hagborg, Ph.D., School Psychologist 
Chatham Central Schoob, Chatham, NY 

Tracking Your School's Success offers educators the step-by-step procedures and practical 
guidance needed to conduct: 

• Sensible assessments and evaluations 

• Record and measure progress 

• Communicate in credible terms what they are trying to accomplish 
and how well they are doing it 

Plus, this comprehensive book instructs the reader on how to configure evaluation strategies 
sensitive to the unique needs, priorities, and goals of the individual schools, and how to 
actually use evaluation information to aid school planning and improve management 
decisions. 

This exceptional book s essential reading for educators involved in site-based evaluation or 
interested in learning how to use evaluation as a tool to solve school problems. 

About the Authors 

Joan L. Herman is Associate Director of the UCLA Center for the Study of 
Evaluation and its Center for Research on Evaluation, Standards, and Student Testing. A 
former elementary school teacher and current Chair cf the Board of Education at Stephen S. 
Wise School, Dr. Herman's research interests focus on the use of evaluation and testing to 
improve schools. 

Lynn Winters is Assessment Director of the Galef Institute, a nonprofit educational 
organization dedicated to serving at-risk student? through arts-based curricular and 
instructional interventions. Dr. Winters is currently directing the development of 
performance arid portfolio assessments for a K-6 arts-infused social studies curriculum. She 
has over 15 years experience in school-level evaluation as both a research director and 
consultant. 

ISBN: 0-8039-6024-7 




